{"id":243,"date":"2016-09-17T18:29:50","date_gmt":"2016-09-18T01:29:50","guid":{"rendered":"http:\/\/blogs.gentoo.org\/zmedico\/?p=243"},"modified":"2016-09-17T18:33:16","modified_gmt":"2016-09-18T01:33:16","slug":"adapting-regular-iterators-to-asynchronous-iterators-in-python","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/zmedico\/2016\/09\/17\/adapting-regular-iterators-to-asynchronous-iterators-in-python\/","title":{"rendered":"Adapting regular iterators to asynchronous iterators in python"},"content":{"rendered":"<p>For I\/O bound tasks, python coroutines make a nice replacement for threads. Unfortunately, there&#8217;s no asynchronous API for reading files, as discussed in the <a href=\"https:\/\/groups.google.com\/forum\/#!topic\/python-tulip\/MvpkQeetWZA\" target=\"_blank\">Best way to read\/write files with AsyncIO<\/a> thread of the python-tulip mailing list.<\/p>\n<p>Meanwhile, it is essential that a long-running coroutine contain some asynchronous calls, since otherwise it will run all the way to completion before any other event loop tasks are allowed to run. For a long-running coroutine that needs to call a conventional iterator (rather than an <a href=\"https:\/\/www.python.org\/dev\/peps\/pep-0492\/#asynchronous-iterators-and-async-for\" target=\"_blank\">asynchronous iterator<\/a>), I&#8217;ve found this converter class to be useful:<\/p>\n<pre>class AsyncIteratorExecutor:\r\n    \"\"\"\r\n    Converts a regular iterator into an asynchronous\r\n    iterator, by executing the iterator in a thread.\r\n    \"\"\"\r\n    def __init__(self, iterator, loop=None, executor=None):\r\n        self.__iterator = iterator\r\n        self.__loop = loop or asyncio.get_event_loop()\r\n        self.__executor = executor\r\n\r\n    def __aiter__(self):\r\n        return self\r\n\r\n    async def __anext__(self):\r\n        value = await self.__loop.run_in_executor(\r\n            self.__executor, next, self.__iterator, self)\r\n        if value is self:\r\n            raise StopAsyncIteration\r\n        return value\r\n<\/pre>\n<p>For example, it can be used to asynchronously read lines of a text file as follows:<\/p>\n<pre>async def cat_file_async(filename):\r\n    with open(filename, 'rt') as f:\r\n        async for line in AsyncIteratorExecutor(f):\r\n            print(line.rstrip())\r\n\r\nif __name__ == '__main__':\r\n    loop = asyncio.get_event_loop()\r\n    try:\r\n        loop.run_until_complete(\r\n            cat_file_async('\/path\/of\/file.txt'))\r\n    finally:\r\n        loop.close()\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>For I\/O bound tasks, python coroutines make a nice replacement for threads. Unfortunately, there&#8217;s no asynchronous API for reading files, as discussed in the Best way to read\/write files with AsyncIO thread of the python-tulip mailing list. Meanwhile, it is essential that a long-running coroutine contain some asynchronous calls, since otherwise it will run all &hellip; <a href=\"https:\/\/blogs.gentoo.org\/zmedico\/2016\/09\/17\/adapting-regular-iterators-to-asynchronous-iterators-in-python\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Adapting regular iterators to asynchronous iterators in python<\/span><\/a><\/p>\n","protected":false},"author":65,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[21],"tags":[23,24,22],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/zmedico\/wp-json\/wp\/v2\/posts\/243"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/zmedico\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/zmedico\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/zmedico\/wp-json\/wp\/v2\/users\/65"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/zmedico\/wp-json\/wp\/v2\/comments?post=243"}],"version-history":[{"count":6,"href":"https:\/\/blogs.gentoo.org\/zmedico\/wp-json\/wp\/v2\/posts\/243\/revisions"}],"predecessor-version":[{"id":249,"href":"https:\/\/blogs.gentoo.org\/zmedico\/wp-json\/wp\/v2\/posts\/243\/revisions\/249"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/zmedico\/wp-json\/wp\/v2\/media?parent=243"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/zmedico\/wp-json\/wp\/v2\/categories?post=243"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/zmedico\/wp-json\/wp\/v2\/tags?post=243"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}