# TextIteratorStreamer compatibility with batch processing

**URL:** <https://discuss.huggingface.co/t/textiteratorstreamer-compatibility-with-batch-processing/46763>\
**Category:** Intermediate\
**Created:** [July 13, 2023, 10:42pm UTC](https://discuss.huggingface.co/t/textiteratorstreamer-compatibility-with-batch-processing/46763 "2023-07-13T22:42:31Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jmurphy97](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@jmurphy97](https://discuss.huggingface.co/u/jmurphy97)\
**Post date:** [July 13, 2023, 10:42pm UTC](https://discuss.huggingface.co/t/textiteratorstreamer-compatibility-with-batch-processing/46763/1 "2023-07-13T22:42:32Z")

</div>

I’m working on a service that can stream LLM responses and I want to make it compatible with batch processing.

Previously I was using the TextIteratorStreamer object to handle the streaming but this is incompatible with batching ([ValueError(“TextStreamer only supports batch size 1”)](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/streamers.py#L87)

Is there any plans on making this feature compatible with batching, or is there any other tool I can use instead?

Would greatly appreciate any advice - cheers!

---

<div class="post-metadata">

**Author:** ![kitaharatomoyo](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@kitaharatomoyo](https://discuss.huggingface.co/u/kitaharatomoyo)\
**Post date:** [July 25, 2023, 6:58am UTC](https://discuss.huggingface.co/t/textiteratorstreamer-compatibility-with-batch-processing/46763/2 "2023-07-25T06:58:50Z")

</div>

you can try this code, I just modified it a little bit:

```auto

class BatchTextIteratorStreamer(TextIteratorStreamer):
    def __init__ (self, batch_size:int, tokenizer: "AutoTokenizer", skip_prompt: bool = False, timeout: Optional[float] = None, **decode_kwargs):
        super(). __init__ (tokenizer, skip_prompt, timeout, **decode_kwargs)
        self.batch_size = batch_size
        self.token_cache = [[] for _ in range(batch_size)]
        self.print_len = [0 for _ in range(batch_size)]
        self.generate_exception = None

    def put(self, value):
        if len(value.shape) != 2:
            value = torch.reshape(value, (self.batch_size, value.shape[0] // self.batch_size))

        if self.skip_prompt and self.next_tokens_are_prompt:
            self.next_tokens_are_prompt = False
            return

        printable_texts = list()
        for idx in range(self.batch_size):
            self.token_cache[idx].extend(value[idx].tolist())
            text = self.tokenizer.decode(self.token_cache[idx], **self.decode_kwargs)

            if text.endswith("\n"):
                printable_text = text[self.print_len[idx] :]
                self.token_cache[idx] = []
                self.print_len[idx] = 0
                # If the last token is a CJK character, we print the characters.
            elif len(text) > 0 and self._is_chinese_char(ord(text[-1])):
                printable_text = text[self.print_len[idx] :]
                self.print_len[idx] += len(printable_text)
            else:
                printable_text = text[self.print_len[idx] : text.rfind(" ") + 1]
                self.print_len[idx] += len(printable_text)
            printable_texts.append(printable_text)

        self.on_finalized_text(printable_texts)

    def end(self):
        printable_texts = list()
        for idx in range(self.batch_size):
            if len(self.token_cache[idx]) > 0:
                text = self.tokenizer.decode(self.token_cache[idx], **self.decode_kwargs)
                printable_text = text[self.print_len[idx] :]
                self.token_cache[idx] = []
                self.print_len[idx] = 0
            else:
                printable_text = ""
            printable_texts.append(printable_text)

        self.next_tokens_are_prompt = True
        self.on_finalized_text(printable_texts, stream_end=True)

    def on_finalized_text(self, texts: List[str], stream_end: bool = False):
        self.text_queue.put(texts, timeout=self.timeout)
        if stream_end:
            self.text_queue.put(self.stop_signal, timeout=self.timeout)

```

---

<div class="post-metadata">

**Author:** ![luv2261](https://avatars.discourse-cdn.com/v4/letter/l/e480ec/32.png) [@luv2261](https://discuss.huggingface.co/u/luv2261)\
**Post date:** [December 4, 2024, 9:32am UTC](https://discuss.huggingface.co/t/textiteratorstreamer-compatibility-with-batch-processing/46763/3 "2024-12-04T09:32:45Z")

</div>

Thanks a lot. It worked for me!!

---

<div class="post-metadata">

**Author:** ![hyperdrive](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hyperdrive/32/37014_2.png) [@hyperdrive](https://discuss.huggingface.co/u/hyperdrive)\
**Post date:** [December 6, 2024, 8:03am UTC](https://discuss.huggingface.co/t/textiteratorstreamer-compatibility-with-batch-processing/46763/4 "2024-12-06T08:03:34Z")

</div>

Pardon my curiosity, but in the spirit of knowledge sharing, can you please elaborate some on why you want a streamer to support batches?