TypeError: 'NoneType' object is not iterable

[sgttwld]

For me the GoogleImageCrawler of icrawler doesn't work anymore. I updated the user agent in crawler.py since that seemed to work in the past, but no luck here. I tried it both on python 3.8 and 3.9 (apple silicon, but shouldn't matter). Again, it worked in the past (like 3-6 months ago).

Even the simple example

from icrawler.builtin import GoogleImageCrawler
searchterm = 'ANY SEARCHTERM'
google_crawler = GoogleImageCrawler(storage={'root_dir': 'test'})
google_crawler.crawl(keyword=searchterm, max_num=1)

gives

...lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File ".../python3.8/site-packages/icrawler-0.6.6-py3.8.egg/icrawler/parser.py", line 104, in worker_exec
    for task in self.parse(response, **kwargs):
TypeError: 'NoneType' object is not iterable

Does anyone know how to fix this, or have the same issue in July 2022?

[Kir-1]

I have the same problem, I tried to roll back to the old version, it does not help. I saw that such a problem was already with this library
i am executing the following code
self.__search_word = 'cat'
self.__count = 10

`google = GoogleImageCrawler(storage={"root_dir": path})
filters = dict(
size='>1024x768',
date=((2020, 1, 1), (2021, 11, 30)))

        google.crawl(keyword=self.__search_word, max_num=self.__count, filters=filters, offset=rnd.randint(0, 500))
    except Exception as _ex:
        logger.error("Something happened when uploading images", _ex)`

at the output I get
2022-07-10 10:36:53,907 - INFO - icrawler.crawler - start crawling...
2022-07-10 10:36:53,907 - INFO - icrawler.crawler - starting 1 feeder threads...
2022-07-10 10:36:53,914 - INFO - feeder - thread feeder-001 exit
2022-07-10 10:36:53,915 - INFO - icrawler.crawler - starting 1 parser threads...
2022-07-10 10:36:53,916 - INFO - icrawler.crawler - starting 1 downloader threads...
2022-07-10 10:36:54,117 - INFO - parser - parsing result page https://www.google.com/search?q=apex&ijn=1&start=150&tbs=isz%3Alt%2Cislt%3Axga%2Ccdr%3A1%2Ccd_min%3A01%2F01%2F2020%2Ccd_max%3A11%2F30%2F2021&tbm=isch
Traceback (most recent call last):
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\Administrator\PycharmProjects\BPG\venv37\lib\site-packages\icrawler\parser.py", line 104, in worker_exec
for task in self.parse(response, **kwargs):
TypeError: 'NoneType' object is not iterable
python-BaseException

[Viachaslau85]

I have the same problem. Works with Bing and Baidu, but does not work with Google. I keep getting the following errors:
2022-07-27 18:52:22,851 - INFO - icrawler.crawler - start crawling... 2022-07-27 18:52:22,852 - INFO - icrawler.crawler - starting 1 feeder threads... 2022-07-27 18:52:22,852 - INFO - icrawler.crawler - starting 1 parser threads... 2022-07-27 18:52:22,853 - INFO - icrawler.crawler - starting 4 downloader threads... 2022-07-27 18:52:23,323 - INFO - parser - parsing result page https://www.google.com/search?q=cat&ijn=0&start=0&tbs=isz%3Al%2Cic%3Aspecific%2Cisc%3Aorange%2Csur%3Afmc%2Ccdr%3A1%2Ccd_min%3A01%2F01%2F2017%2Ccd_max%3A11%2F30%2F2017&tbm=isch Exception in thread parser-001: Traceback (most recent call last): File "C:\Python310\lib\threading.py", line 1009, in _bootstrap_inner self.run() File "C:\Python310\lib\threading.py", line 946, in run self._target(*self._args, **self._kwargs) File "C:\Python310\lib\site-packages\icrawler\parser.py", line 104, in worker_exec for task in self.parse(response, **kwargs): TypeError: 'NoneType' object is not iterable 2022-07-27 18:52:27,857 - INFO - downloader - no more download task for thread downloader-001 2022-07-27 18:52:27,858 - INFO - downloader - thread downloader-001 exit 2022-07-27 18:52:27,858 - INFO - downloader - no more download task for thread downloader-003 2022-07-27 18:52:27,858 - INFO - downloader - thread downloader-003 exit 2022-07-27 18:52:27,858 - INFO - downloader - no more download task for thread downloader-004 2022-07-27 18:52:27,858 - INFO - downloader - thread downloader-004 exit 2022-07-27 18:52:27,859 - INFO - downloader - no more download task for thread downloader-002 2022-07-27 18:52:27,859 - INFO - downloader - thread downloader-002 exit 2022-07-27 18:52:27,894 - INFO - icrawler.crawler - Crawling task done!

[feay1234]

got the same problem.

[dravicenna]

The same problem

[philborman]

The problem is in builtin/google.py replace the parse function around line 148 with this...

def parse(self, response):
    soup = BeautifulSoup(
        response.content.decode('utf-8', 'ignore'), 'lxml')
    images = soup.find_all(name='img')
    uris = []
    for img in images:
        if img.has_attr('src'):
            uris.append(img['src'])
    return [{'file_url': uri} for uri in uris]

[Viachaslau85]

The problem is in builtin/google.py replace the parse function around line 148 with this...

def parse(self, response):
    soup = BeautifulSoup(
        response.content.decode('utf-8', 'ignore'), 'lxml')
    images = soup.find_all(name='img')
    uris = []
    for img in images:
        if img.has_attr('src'):
            uris.append(img['src'])
    return [{'file_url': uri} for uri in uris]

Much better, but it still doesn't work. It generates errors of the following type:

2022-12-15 09:27:02,994 - ERROR - downloader - Exception caught when downloading file //www.gstatic.com/images/branding/googlelogo/svg/googlelogo_clr_160x56px.svg, error: '', remaining retry times: 2
2022-12-15 09:27:02,996 - ERROR - downloader - Exception caught when downloading file //www.gstatic.com/images/branding/googlelogo/svg/googlelogo_clr_160x56px.svg, error: '', remaining retry times: 1

[jfreyberg]

I can confirm the comments from @philborman and @Viachaslau85. This particular issue seems to be solved by using the provided code snippet (use my fork to pull the relevant changes: pip install git+git://github.com/jfreyberg/icrawler@master --upgrade) but it yields the new downloading error.

[philborman]

The downloading error is easy to fix, Just add this line before uris.append if 'images/branding/' not in img['src']:

…

On Mon, 19 Dec 2022, 20:45 Julian Freyberg, ***@***.***> wrote: I can confirm the comments from @philborman <https://github.com/philborman> and @Viachaslau85 <https://github.com/Viachaslau85>. This particular issue seems to be solved by using the provided code snippet (use my fork to pull the relevant changes: pip install ***@***.*** --upgrade) but it yields the new downloading error. — Reply to this email directly, view it on GitHub <#107 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC4YOOIQFMFLBPJXVQ7IY3TWOC3O5ANCNFSM53D4UUDA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jfreyberg]

Thank you @philborman!
I had to modify it some more for the thing to work:

if 'images/branding/' not in img['src']:
    img_src = img['src']
    if not img_src.startswith('http'):
        img_src = 'https:' + img_src
    uris.append(img_src)

Somehow my URLs lacked the protocol.

I did not properly test this (I can only confirm it worked for google images) so I can not create a pull request, but if anyone wants to use my repo to fix this, feel free: https://github.com/jfreyberg/icrawler