[{"data":1,"prerenderedAt":1329},["ShallowReactive",2],{"post-using-python-multiprocessing-to-optimize-puzzle-solving":3},{"id":4,"title":5,"body":6,"canonicalUrl":1314,"cover":1315,"date":1316,"description":1317,"draft":1318,"extension":1319,"hashnodeId":1320,"meta":1321,"navigation":176,"path":1322,"seo":1323,"slug":1324,"stem":1324,"tags":1325,"__hash__":1328},"posts\u002Fusing-python-multiprocessing-to-optimize-puzzle-solving.md","Using Python multiprocessing to optimize a problem",{"type":7,"value":8,"toc":1299},"minimark",[9,14,32,35,39,42,49,103,106,114,117,120,124,127,140,147,653,662,665,674,684,688,698,704,707,752,755,761,768,771,782,785,791,795,806,814,835,1213,1216,1222,1225,1229,1236,1242,1247,1250,1256,1259,1262,1265,1269,1283,1286,1289,1292,1295],[10,11,13],"h2",{"id":12},"introduction","Introduction",[15,16,17,18,25,26,31],"p",{},"This is the final part of the Python puzzle game series. ",[19,20,24],"a",{"href":21,"rel":22},"https:\u002F\u002Frajeev.dev\u002Fcreating-puzzle-game-using-python-turtle-1",[23],"nofollow","In the first part"," we learnt to create a tiles puzzle game using Turtle module. And then ",[19,27,30],{"href":28,"rel":29},"https:\u002F\u002Frajeev.dev\u002Fsolve-puzzle-game-using-python",[23],"in the second part"," created an algorithm to find a solution of the puzzle. But the algorithm can be optimized further, and in this article we will look at some of the ways we can achieve that.",[15,33,34],{},"To make sense of this article, I suggest that first you go through the previous posts in this series (if you haven't already done so).",[10,36,38],{"id":37},"the-need-for-further-optimization","The need for further optimization",[15,40,41],{},"Let's run the final code from the previous post on a different puzzle which requires more number of moves.",[15,43,44],{},[45,46],"img",{"alt":47,"src":48},"A different puzzle","\u002Fimages\u002Fposts\u002Fusing-python-multiprocessing-to-optimize-puzzle-solving\u002FRMHO7pz6T-088b45b524.png",[50,51,56],"pre",{"className":52,"code":53,"language":54,"meta":55,"style":55},"language-python shiki shiki-themes github-light github-dark","play([\n    ['hot pink', 'turquoise', 'yellow', 'white', 'turquoise'],\n    ['white', 'hot pink', 'turquoise', 'yellow', 'hot pink'],\n    ['white', 'yellow', 'hot pink', 'white', 'yellow'],\n    ['hot pink', 'yellow', 'white', 'hot pink', 'white'],\n    ['turquoise', 'turquoise', 'yellow', 'hot pink', 'yellow']\n])\n","python","",[57,58,59,67,73,79,85,91,97],"code",{"__ignoreMap":55},[60,61,64],"span",{"class":62,"line":63},"line",1,[60,65,66],{},"play([\n",[60,68,70],{"class":62,"line":69},2,[60,71,72],{},"    ['hot pink', 'turquoise', 'yellow', 'white', 'turquoise'],\n",[60,74,76],{"class":62,"line":75},3,[60,77,78],{},"    ['white', 'hot pink', 'turquoise', 'yellow', 'hot pink'],\n",[60,80,82],{"class":62,"line":81},4,[60,83,84],{},"    ['white', 'yellow', 'hot pink', 'white', 'yellow'],\n",[60,86,88],{"class":62,"line":87},5,[60,89,90],{},"    ['hot pink', 'yellow', 'white', 'hot pink', 'white'],\n",[60,92,94],{"class":62,"line":93},6,[60,95,96],{},"    ['turquoise', 'turquoise', 'yellow', 'hot pink', 'yellow']\n",[60,98,100],{"class":62,"line":99},7,[60,101,102],{},"])\n",[15,104,105],{},"These are the results",[50,107,112],{"className":108,"code":110,"language":111},[109],"language-text","start time: 0.113825565\nchanged min_moves to: 16, 4000001111223334, time: 0.12959095\nchanged min_moves to: 15, 400000114313122, time: 0.135476449\nchanged min_moves to: 14, 40000432310211, time: 0.483449261\nchanged min_moves to: 13, 4433100000122, time: 49.005072394\nNo more jobs: final count: 2012130, time: 324.622757167\nresult of operation: {'min_moves': '4433100000122', 'min_moves_len': 13, 'count': 2012130}\n","text",[57,113,110],{"__ignoreMap":55},[15,115,116],{},"So even with our previous optimization, we ran close to 2 million end to end sequences, and it took us around 5 minutes and 25 seconds to finish the job. Do remember that we're only looking for one valid solution, and not trying to find all possible sequences of the same length (and believe me, there will be many such sequences).",[15,118,119],{},"So, can we do something more?",[121,122],"media-embed",{"url":123},"https:\u002F\u002Fmedia.giphy.com\u002Fmedia\u002Fia3zQtyPYYhcpznFJ2\u002Fgiphy.gif",[15,125,126],{},"Well, what if we could run some of these jobs in parallel? That should save us some time, don't you think? Let's find out.",[128,129,131,132,135,136,139],"h3",{"id":130},"modified-play-find_min_moves-functions","Modified ",[57,133,134],{},"play"," & ",[57,137,138],{},"find_min_moves"," functions",[15,141,142,143,146],{},"We will be using ",[57,144,145],{},"multiprocessing"," module to run multiple processes in parallel to see if it can make a difference. The reason for going with multiple processes instead of multiple threads is simple, our code is pure compute problem with no IO related tasks. Multiple threads make sense where there are some wait times involved (as in the case of IO tasks, like downloading something from internet etc).",[50,148,150],{"className":52,"code":149,"language":54,"meta":55,"style":55},"from timeit import default_timer as timer\nimport copy\nimport multiprocessing as mp\nimport os\n\nimport AutoPlay\n\ndef find_min_moves(task):\n    # Create an internal job list for this process and add the incoming task to it\n    jobs = [task]\n    pid = os.getpid()\n    result = {'min_moves': None, 'min_moves_len': 0, 'count': 0, 'pid': pid}\n\n    print(\n        f'entered pid: {pid}: for moves: {task[\"curr_move\"]}, time: {timer()}')\n\n    while True:\n        # See if we've any jobs left. If there is no job, break the loop\n        job = jobs.pop() if len(jobs) > 0 else None\n        if job is None:\n            print(\n                f'No more jobs: pid: {pid}, final count: {result[\"count\"]}, time: {timer()}')\n            break\n\n        # Handle the current job. This will take of the combinations till its logical\n        # end (until the board is clear). Other encountered combinations will be added\n        # to the job list for processing in due course\n        final_moves_seq = AutoPlay.handle_job_recurse(\n            job, jobs, result['min_moves_len'])\n\n        result['count'] += 1\n\n        # If the one processed combination has minimum length, then that is the minimum\n        # numbers of moves needed to solve the puzzle\n        if result['min_moves_len'] == 0 or (final_moves_seq is not None\n                                            and len(final_moves_seq) \u003C result['min_moves_len']):\n            result['min_moves'] = final_moves_seq\n            result['min_moves_len'] = len(final_moves_seq)\n            print(\n                f'pid: {pid}, changed min_moves to: {result[\"min_moves_len\"]}, {final_moves_seq}, time: {timer()}')\n\n    return result\n\ndef play(colors):\n    # Single game object which holds the tiles in play,\n    # and the current connections groups and clickables\n    game = {\n        'tiles': [],\n        'clickables': [],\n        'connection_groups': []\n    }\n\n    # Set the board as per the input colors\n    for col in range(AutoPlay.MAX_COLS):\n        game['tiles'].append([])\n        for row in range(AutoPlay.MAX_ROWS):\n            tile = {'id': (row, col), \"connections\": [],\n                    \"clickable\": False, \"color\": colors[col][row]}\n            game['tiles'][col].append(tile)\n\n    # Go through the tiles and find out the connections\n    # between them, and also save the clickables\n    for col in range(AutoPlay.MAX_COLS):\n        AutoPlay.process_tile(game, 0, col)\n\n    start = timer()\n    print(f'start time: {start}')\n\n    # Create as many tasks as there are connection groups.\n    # We're using deepcopy to create a deeply cloned game\n    # object for each task. The current move is the first\n    # entry of every connection group (the lowest column\n    # index in the bottom row)\n    tasks = []\n    for connections in game['connection_groups']:\n        g = copy.deepcopy(game)\n        tasks.append(\n            {'game': g, 'curr_move': connections[0], 'past_moves': None})\n\n    # Get a managed pool from multiprocessing, and distribute the tasks to these\n    # pools. By default it will create processes equal to the number returned by\n    # os.cpu_count().\n    with mp.Pool() as pool:\n        results = pool.map(find_min_moves, tasks)\n        print('got results:', timer())\n        for result in results:\n            print('result:', result)\n",[57,151,152,157,162,167,172,178,183,187,193,199,205,211,217,222,228,234,239,245,251,257,263,269,275,281,286,292,298,304,310,316,321,327,332,338,344,350,356,362,368,373,379,384,390,395,401,407,413,419,425,431,437,443,448,454,460,466,472,478,484,490,495,501,507,512,518,523,529,535,540,546,552,558,564,570,576,582,588,594,600,605,611,617,623,629,635,641,647],{"__ignoreMap":55},[60,153,154],{"class":62,"line":63},[60,155,156],{},"from timeit import default_timer as timer\n",[60,158,159],{"class":62,"line":69},[60,160,161],{},"import copy\n",[60,163,164],{"class":62,"line":75},[60,165,166],{},"import multiprocessing as mp\n",[60,168,169],{"class":62,"line":81},[60,170,171],{},"import os\n",[60,173,174],{"class":62,"line":87},[60,175,177],{"emptyLinePlaceholder":176},true,"\n",[60,179,180],{"class":62,"line":93},[60,181,182],{},"import AutoPlay\n",[60,184,185],{"class":62,"line":99},[60,186,177],{"emptyLinePlaceholder":176},[60,188,190],{"class":62,"line":189},8,[60,191,192],{},"def find_min_moves(task):\n",[60,194,196],{"class":62,"line":195},9,[60,197,198],{},"    # Create an internal job list for this process and add the incoming task to it\n",[60,200,202],{"class":62,"line":201},10,[60,203,204],{},"    jobs = [task]\n",[60,206,208],{"class":62,"line":207},11,[60,209,210],{},"    pid = os.getpid()\n",[60,212,214],{"class":62,"line":213},12,[60,215,216],{},"    result = {'min_moves': None, 'min_moves_len': 0, 'count': 0, 'pid': pid}\n",[60,218,220],{"class":62,"line":219},13,[60,221,177],{"emptyLinePlaceholder":176},[60,223,225],{"class":62,"line":224},14,[60,226,227],{},"    print(\n",[60,229,231],{"class":62,"line":230},15,[60,232,233],{},"        f'entered pid: {pid}: for moves: {task[\"curr_move\"]}, time: {timer()}')\n",[60,235,237],{"class":62,"line":236},16,[60,238,177],{"emptyLinePlaceholder":176},[60,240,242],{"class":62,"line":241},17,[60,243,244],{},"    while True:\n",[60,246,248],{"class":62,"line":247},18,[60,249,250],{},"        # See if we've any jobs left. If there is no job, break the loop\n",[60,252,254],{"class":62,"line":253},19,[60,255,256],{},"        job = jobs.pop() if len(jobs) > 0 else None\n",[60,258,260],{"class":62,"line":259},20,[60,261,262],{},"        if job is None:\n",[60,264,266],{"class":62,"line":265},21,[60,267,268],{},"            print(\n",[60,270,272],{"class":62,"line":271},22,[60,273,274],{},"                f'No more jobs: pid: {pid}, final count: {result[\"count\"]}, time: {timer()}')\n",[60,276,278],{"class":62,"line":277},23,[60,279,280],{},"            break\n",[60,282,284],{"class":62,"line":283},24,[60,285,177],{"emptyLinePlaceholder":176},[60,287,289],{"class":62,"line":288},25,[60,290,291],{},"        # Handle the current job. This will take of the combinations till its logical\n",[60,293,295],{"class":62,"line":294},26,[60,296,297],{},"        # end (until the board is clear). Other encountered combinations will be added\n",[60,299,301],{"class":62,"line":300},27,[60,302,303],{},"        # to the job list for processing in due course\n",[60,305,307],{"class":62,"line":306},28,[60,308,309],{},"        final_moves_seq = AutoPlay.handle_job_recurse(\n",[60,311,313],{"class":62,"line":312},29,[60,314,315],{},"            job, jobs, result['min_moves_len'])\n",[60,317,319],{"class":62,"line":318},30,[60,320,177],{"emptyLinePlaceholder":176},[60,322,324],{"class":62,"line":323},31,[60,325,326],{},"        result['count'] += 1\n",[60,328,330],{"class":62,"line":329},32,[60,331,177],{"emptyLinePlaceholder":176},[60,333,335],{"class":62,"line":334},33,[60,336,337],{},"        # If the one processed combination has minimum length, then that is the minimum\n",[60,339,341],{"class":62,"line":340},34,[60,342,343],{},"        # numbers of moves needed to solve the puzzle\n",[60,345,347],{"class":62,"line":346},35,[60,348,349],{},"        if result['min_moves_len'] == 0 or (final_moves_seq is not None\n",[60,351,353],{"class":62,"line":352},36,[60,354,355],{},"                                            and len(final_moves_seq) \u003C result['min_moves_len']):\n",[60,357,359],{"class":62,"line":358},37,[60,360,361],{},"            result['min_moves'] = final_moves_seq\n",[60,363,365],{"class":62,"line":364},38,[60,366,367],{},"            result['min_moves_len'] = len(final_moves_seq)\n",[60,369,371],{"class":62,"line":370},39,[60,372,268],{},[60,374,376],{"class":62,"line":375},40,[60,377,378],{},"                f'pid: {pid}, changed min_moves to: {result[\"min_moves_len\"]}, {final_moves_seq}, time: {timer()}')\n",[60,380,382],{"class":62,"line":381},41,[60,383,177],{"emptyLinePlaceholder":176},[60,385,387],{"class":62,"line":386},42,[60,388,389],{},"    return result\n",[60,391,393],{"class":62,"line":392},43,[60,394,177],{"emptyLinePlaceholder":176},[60,396,398],{"class":62,"line":397},44,[60,399,400],{},"def play(colors):\n",[60,402,404],{"class":62,"line":403},45,[60,405,406],{},"    # Single game object which holds the tiles in play,\n",[60,408,410],{"class":62,"line":409},46,[60,411,412],{},"    # and the current connections groups and clickables\n",[60,414,416],{"class":62,"line":415},47,[60,417,418],{},"    game = {\n",[60,420,422],{"class":62,"line":421},48,[60,423,424],{},"        'tiles': [],\n",[60,426,428],{"class":62,"line":427},49,[60,429,430],{},"        'clickables': [],\n",[60,432,434],{"class":62,"line":433},50,[60,435,436],{},"        'connection_groups': []\n",[60,438,440],{"class":62,"line":439},51,[60,441,442],{},"    }\n",[60,444,446],{"class":62,"line":445},52,[60,447,177],{"emptyLinePlaceholder":176},[60,449,451],{"class":62,"line":450},53,[60,452,453],{},"    # Set the board as per the input colors\n",[60,455,457],{"class":62,"line":456},54,[60,458,459],{},"    for col in range(AutoPlay.MAX_COLS):\n",[60,461,463],{"class":62,"line":462},55,[60,464,465],{},"        game['tiles'].append([])\n",[60,467,469],{"class":62,"line":468},56,[60,470,471],{},"        for row in range(AutoPlay.MAX_ROWS):\n",[60,473,475],{"class":62,"line":474},57,[60,476,477],{},"            tile = {'id': (row, col), \"connections\": [],\n",[60,479,481],{"class":62,"line":480},58,[60,482,483],{},"                    \"clickable\": False, \"color\": colors[col][row]}\n",[60,485,487],{"class":62,"line":486},59,[60,488,489],{},"            game['tiles'][col].append(tile)\n",[60,491,493],{"class":62,"line":492},60,[60,494,177],{"emptyLinePlaceholder":176},[60,496,498],{"class":62,"line":497},61,[60,499,500],{},"    # Go through the tiles and find out the connections\n",[60,502,504],{"class":62,"line":503},62,[60,505,506],{},"    # between them, and also save the clickables\n",[60,508,510],{"class":62,"line":509},63,[60,511,459],{},[60,513,515],{"class":62,"line":514},64,[60,516,517],{},"        AutoPlay.process_tile(game, 0, col)\n",[60,519,521],{"class":62,"line":520},65,[60,522,177],{"emptyLinePlaceholder":176},[60,524,526],{"class":62,"line":525},66,[60,527,528],{},"    start = timer()\n",[60,530,532],{"class":62,"line":531},67,[60,533,534],{},"    print(f'start time: {start}')\n",[60,536,538],{"class":62,"line":537},68,[60,539,177],{"emptyLinePlaceholder":176},[60,541,543],{"class":62,"line":542},69,[60,544,545],{},"    # Create as many tasks as there are connection groups.\n",[60,547,549],{"class":62,"line":548},70,[60,550,551],{},"    # We're using deepcopy to create a deeply cloned game\n",[60,553,555],{"class":62,"line":554},71,[60,556,557],{},"    # object for each task. The current move is the first\n",[60,559,561],{"class":62,"line":560},72,[60,562,563],{},"    # entry of every connection group (the lowest column\n",[60,565,567],{"class":62,"line":566},73,[60,568,569],{},"    # index in the bottom row)\n",[60,571,573],{"class":62,"line":572},74,[60,574,575],{},"    tasks = []\n",[60,577,579],{"class":62,"line":578},75,[60,580,581],{},"    for connections in game['connection_groups']:\n",[60,583,585],{"class":62,"line":584},76,[60,586,587],{},"        g = copy.deepcopy(game)\n",[60,589,591],{"class":62,"line":590},77,[60,592,593],{},"        tasks.append(\n",[60,595,597],{"class":62,"line":596},78,[60,598,599],{},"            {'game': g, 'curr_move': connections[0], 'past_moves': None})\n",[60,601,603],{"class":62,"line":602},79,[60,604,177],{"emptyLinePlaceholder":176},[60,606,608],{"class":62,"line":607},80,[60,609,610],{},"    # Get a managed pool from multiprocessing, and distribute the tasks to these\n",[60,612,614],{"class":62,"line":613},81,[60,615,616],{},"    # pools. By default it will create processes equal to the number returned by\n",[60,618,620],{"class":62,"line":619},82,[60,621,622],{},"    # os.cpu_count().\n",[60,624,626],{"class":62,"line":625},83,[60,627,628],{},"    with mp.Pool() as pool:\n",[60,630,632],{"class":62,"line":631},84,[60,633,634],{},"        results = pool.map(find_min_moves, tasks)\n",[60,636,638],{"class":62,"line":637},85,[60,639,640],{},"        print('got results:', timer())\n",[60,642,644],{"class":62,"line":643},86,[60,645,646],{},"        for result in results:\n",[60,648,650],{"class":62,"line":649},87,[60,651,652],{},"            print('result:', result)\n",[50,654,656],{"className":52,"code":655,"language":54,"meta":55,"style":55},"with mp.Pool() as pool:\n",[57,657,658],{"__ignoreMap":55},[60,659,660],{"class":62,"line":63},[60,661,655],{},[15,663,664],{},"gives us a managed pool which cleans after itself.",[50,666,668],{"className":52,"code":667,"language":54,"meta":55,"style":55},"results = pool.map(find_min_moves, tasks)\n",[57,669,670],{"__ignoreMap":55},[60,671,672],{"class":62,"line":63},[60,673,667],{},[15,675,676,677,679,680,683],{},"is a blocking function which takes care of distributing the tasks to the processes from the pool. For every process, ",[57,678,138],{}," function is called with one task from the ",[57,681,682],{},"tasks"," list.",[128,685,687],{"id":686},"the-result","The result",[15,689,690,691,693,694,697],{},"If we try to call our ",[57,692,134],{}," function now, we will get the below ",[57,695,696],{},"RuntimeError"," (on Windows and macOS).",[50,699,702],{"className":700,"code":701,"language":111},[109],"An attempt has been made to start a new process before the\ncurrent process has finished its bootstrapping phase.\n \nThis probably means that you are not using fork to start your\nchild processes and you have forgotten to use the proper idiom\nin the main module:\n \n    if __name__ == '__main__':\n        freeze_support()\n        ...\n \nThe \"freeze_support()\" line can be omitted if the program\nis not going to be frozen to produce an executable.\n",[57,703,701],{"__ignoreMap":55},[15,705,706],{},"As the error is saying, we need to protect the entry point of our program, else it will enter into an endless loop while spawning child processes. Let's modify the calling part",[50,708,710],{"className":52,"code":709,"language":54,"meta":55,"style":55},"if __name__ == '__main__':\n    play([\n        ['hot pink', 'turquoise', 'yellow', 'white', 'turquoise'],\n        ['white', 'hot pink', 'turquoise', 'yellow', 'hot pink'],\n        ['white', 'yellow', 'hot pink', 'white', 'yellow'],\n        ['hot pink', 'yellow', 'white', 'hot pink', 'white'],\n        ['turquoise', 'turquoise', 'yellow', 'hot pink', 'yellow']\n    ])\n",[57,711,712,717,722,727,732,737,742,747],{"__ignoreMap":55},[60,713,714],{"class":62,"line":63},[60,715,716],{},"if __name__ == '__main__':\n",[60,718,719],{"class":62,"line":69},[60,720,721],{},"    play([\n",[60,723,724],{"class":62,"line":75},[60,725,726],{},"        ['hot pink', 'turquoise', 'yellow', 'white', 'turquoise'],\n",[60,728,729],{"class":62,"line":81},[60,730,731],{},"        ['white', 'hot pink', 'turquoise', 'yellow', 'hot pink'],\n",[60,733,734],{"class":62,"line":87},[60,735,736],{},"        ['white', 'yellow', 'hot pink', 'white', 'yellow'],\n",[60,738,739],{"class":62,"line":93},[60,740,741],{},"        ['hot pink', 'yellow', 'white', 'hot pink', 'white'],\n",[60,743,744],{"class":62,"line":99},[60,745,746],{},"        ['turquoise', 'turquoise', 'yellow', 'hot pink', 'yellow']\n",[60,748,749],{"class":62,"line":189},[60,750,751],{},"    ])\n",[15,753,754],{},"And here are the results of the execution",[50,756,759],{"className":757,"code":758,"language":111},[109],"start time: 0.05643949\nentered pid: 39066: for moves: (0, 0), time: 0.120805656\npid: 39066, changed min_moves to: 18, 000001111223334444, time: 0.128779476\npid: 39066, changed min_moves to: 17, 00000111122344334, time: 0.130322022\npid: 39066, changed min_moves to: 16, 0000011112423334, time: 0.131969189\npid: 39066, changed min_moves to: 15, 000001144313122, time: 0.209503056\nentered pid: 39067: for moves: (0, 1), time: 0.193182385\npid: 39067, changed min_moves to: 16, 1000001223334444, time: 0.203432333\npid: 39067, changed min_moves to: 15, 100000122344334, time: 0.206294855\npid: 39067, changed min_moves to: 14, 10000012423334, time: 0.209911962\nentered pid: 39068: for moves: (0, 3), time: 0.2225178\nentered pid: 39069: for moves: (0, 4), time: 0.22732867\npid: 39068, changed min_moves to: 18, 300000111122334444, time: 0.254337267\npid: 39068, changed min_moves to: 17, 30000011112244334, time: 0.268903578\npid: 39069, changed min_moves to: 16, 4000001111223334, time: 0.272185431\npid: 39069, changed min_moves to: 15, 400000114313122, time: 0.278042101\npid: 39068, changed min_moves to: 16, 3000001114431224, time: 0.337434568\npid: 39068, changed min_moves to: 15, 300000114131224, time: 0.351929327\npid: 39069, changed min_moves to: 14, 40000432310211, time: 1.06737366\npid: 39068, changed min_moves to: 14, 30000423104211, time: 1.918205703\npid: 39067, changed min_moves to: 13, 1004430003122, time: 2.159644669\npid: 39066, changed min_moves to: 14, 00004432310211, time: 4.574963226\nNo more jobs: pid: 39067, final count: 171415, time: 52.145841877\npid: 39069, changed min_moves to: 13, 4433100000122, time: 96.061340938\nNo more jobs: pid: 39069, final count: 418533, time: 115.766546083\npid: 39068, changed min_moves to: 13, 3431000001224, time: 217.433144061\nNo more jobs: pid: 39068, final count: 1166097, time: 256.900814533\nNo more jobs: pid: 39066, final count: 2631991, time: 479.720000441\ngot results: 479.826363948\nresult: {'min_moves': '00004432310211', 'min_moves_len': 14, 'count': 2631991, 'pid': 39066}\nresult: {'min_moves': '1004430003122', 'min_moves_len': 13, 'count': 171415, 'pid': 39067}\nresult: {'min_moves': '3431000001224', 'min_moves_len': 13, 'count': 1166097, 'pid': 39068}\nresult: {'min_moves': '4433100000122', 'min_moves_len': 13, 'count': 418533, 'pid': 39069}\n",[57,760,758],{"__ignoreMap":55},[15,762,763,767],{},[764,765,766],"strong",{},"Wow! This is bad..."," We took close to 8 minutes, to finish executing the program, and did almost 4.4 million end to end sequences. Please notice that the program was running in parallel for sometime with process ids 39066-39069. We found 3 different sequences all giving us 13 as the minimum number of moves.",[15,769,770],{},"So what went wrong as compared to the previous case?",[15,772,773,774,777,778,781],{},"The thing is, in the case of a single process, the ",[57,775,776],{},"min_moves"," variable was same for all the executions, but here that is not the case. We had different variables (",[57,779,780],{},"result[\"min_moves_len\"]",") for different starting moves. So, even though the code ran in parallel, individually every process did more end-to-end sequences because of the forced silos.",[15,783,784],{},"**How do we fix this? **",[15,786,787,788,790],{},"We somehow need to have a common ",[57,789,776],{}," variable between these processes. We can't simply use a global variable for the same as different processes have their own memory space. We need to take help from multiprocessing module itself for achieving data sharing.",[10,792,794],{"id":793},"the-final-optimization","The final optimization",[15,796,797,798,801,802,805],{},"We will use a synchronized shared object ",[57,799,800],{},"Value()"," for storing the min_moves_len, and then we will use it in our processes to make a decision. Since knowing the current min moves length in our processes is sufficient for us, and so, using ",[57,803,804],{},"Value"," makes more sense (for storing a number, and it is faster too) as compared to other ways of sharing data between processes.",[128,807,131,809,811,812,139],{"id":808},"modified-play-and-find_min_moves-functions",[57,810,134],{}," and ",[57,813,138],{},[15,815,816,817,820,821,823,824,826,827,830,831,834],{},"We are using another function called ",[57,818,819],{},"init_globals"," to initialize each process with a ",[57,822,776],{}," global variable for that process. We need to pass this function as initializer while creating the processes pool. We also create one ",[57,825,800],{}," object (",[57,828,829],{},"min_val = mp.Value('i', 0)",", where 'i' denotes a signed integer), and pass that to the initializer function as ",[57,832,833],{},"initargs",".",[50,836,838],{"className":52,"code":837,"language":54,"meta":55,"style":55},"def init_globals(min_val):\n    global min_moves\n    min_moves = min_val\n\n# To use the min_moves variable we just need to use its `value` attribute\n# like, min_moves.value\ndef find_min_moves(task):\n    # Create an interal job list for this process and add the incoming task to it\n    jobs = [task]\n    pid = os.getpid()\n    result = {'min_moves': None, 'min_moves_len': 0, 'count': 0, 'pid': pid}\n\n    print(\n        f'entered pid: {pid}: for moves: {task[\"curr_move\"]}, time: {timer()}')\n\n    while True:\n        # See if we've any jobs left. If there is no job, break the loop\n        job = jobs.pop() if len(jobs) > 0 else None\n        if job is None:\n            print(\n                f'No more jobs: pid: {pid}, final count: {result[\"count\"]}, time: {timer()}')\n            break\n\n        # Handle the current job. This will take of the combinations till its logical\n        # end (until the board is clear). Other encountered combinations will be added\n        # to the job list for processing in due course\n        final_moves_seq = AutoPlay.handle_job_recurse(\n            job, jobs, min_moves.value)\n\n        result['count'] += 1\n\n        # If the one processed combination has minimum length, then that is the minimum\n        # numbers of moves needed to solve the puzzle\n        if min_moves.value == 0 or (final_moves_seq is not None\n                                    and len(final_moves_seq) \u003C min_moves.value):\n            min_moves.value = len(final_moves_seq)\n            result['min_moves'] = final_moves_seq\n            result['min_moves_len'] = min_moves.value\n            print(\n                f'pid: {pid}, changed min_moves to: {result[\"min_moves_len\"]}, {final_moves_seq}, time: {timer()}')\n\n    return result\n\ndef play(colors):\n    # Single game object which holds the tiles in play,\n    # and the current connections groups and clickables\n    game = {\n        'tiles': [],\n        'clickables': [],\n        'connection_groups': []\n    }\n\n    # Set the board as per the input colors\n    for col in range(AutoPlay.MAX_COLS):\n        game['tiles'].append([])\n        for row in range(AutoPlay.MAX_ROWS):\n            tile = {'id': (row, col), \"connections\": [],\n                    \"clickable\": False, \"color\": colors[col][row]}\n            game['tiles'][col].append(tile)\n\n    # Go through the tiles and find out the connections\n    # between them, and also save the clickables\n    for col in range(AutoPlay.MAX_COLS):\n        AutoPlay.process_tile(game, 0, col)\n\n    start = timer()\n    print(f'start time: {start}')\n\n    # Create as many tasks as there are connection groups.\n    # We're using deepcopy to create a deeply cloned game\n    # object for each task. The current move is the first\n    # entry of every connection group (the lowest column\n    # index in the bottom row)\n    tasks = []\n    for connections in game['connection_groups']:\n        g = copy.deepcopy(game)\n        tasks.append(\n            {'game': g, 'curr_move': connections[0], 'past_moves': None})\n\n    min_val = mp.Value('i', 0)\n    # Get a managed pool from multiprocessing, and distribute the tasks to these\n    # pools. By default it will create processes equal to the number returned by\n    # os.cpu_count(). Also initialize min_moves global for each process using a \n    # Value() shared object\n    with mp.Pool(initializer=init_globals, initargs=(min_val,)) as pool:\n        results = pool.map(find_min_moves, tasks)\n        print('got results:', timer())\n        for result in results:\n            print('result:', result)\n",[57,839,840,845,850,855,859,864,869,873,878,882,886,890,894,898,902,906,910,914,918,922,926,930,934,938,942,946,950,954,959,963,967,971,975,979,984,989,994,998,1003,1007,1011,1015,1019,1023,1027,1031,1035,1039,1043,1047,1051,1055,1059,1063,1067,1071,1075,1079,1083,1087,1091,1095,1099,1103,1107,1111,1115,1119,1123,1127,1131,1135,1139,1143,1147,1151,1155,1159,1163,1167,1172,1176,1180,1185,1190,1195,1199,1203,1208],{"__ignoreMap":55},[60,841,842],{"class":62,"line":63},[60,843,844],{},"def init_globals(min_val):\n",[60,846,847],{"class":62,"line":69},[60,848,849],{},"    global min_moves\n",[60,851,852],{"class":62,"line":75},[60,853,854],{},"    min_moves = min_val\n",[60,856,857],{"class":62,"line":81},[60,858,177],{"emptyLinePlaceholder":176},[60,860,861],{"class":62,"line":87},[60,862,863],{},"# To use the min_moves variable we just need to use its `value` attribute\n",[60,865,866],{"class":62,"line":93},[60,867,868],{},"# like, min_moves.value\n",[60,870,871],{"class":62,"line":99},[60,872,192],{},[60,874,875],{"class":62,"line":189},[60,876,877],{},"    # Create an interal job list for this process and add the incoming task to it\n",[60,879,880],{"class":62,"line":195},[60,881,204],{},[60,883,884],{"class":62,"line":201},[60,885,210],{},[60,887,888],{"class":62,"line":207},[60,889,216],{},[60,891,892],{"class":62,"line":213},[60,893,177],{"emptyLinePlaceholder":176},[60,895,896],{"class":62,"line":219},[60,897,227],{},[60,899,900],{"class":62,"line":224},[60,901,233],{},[60,903,904],{"class":62,"line":230},[60,905,177],{"emptyLinePlaceholder":176},[60,907,908],{"class":62,"line":236},[60,909,244],{},[60,911,912],{"class":62,"line":241},[60,913,250],{},[60,915,916],{"class":62,"line":247},[60,917,256],{},[60,919,920],{"class":62,"line":253},[60,921,262],{},[60,923,924],{"class":62,"line":259},[60,925,268],{},[60,927,928],{"class":62,"line":265},[60,929,274],{},[60,931,932],{"class":62,"line":271},[60,933,280],{},[60,935,936],{"class":62,"line":277},[60,937,177],{"emptyLinePlaceholder":176},[60,939,940],{"class":62,"line":283},[60,941,291],{},[60,943,944],{"class":62,"line":288},[60,945,297],{},[60,947,948],{"class":62,"line":294},[60,949,303],{},[60,951,952],{"class":62,"line":300},[60,953,309],{},[60,955,956],{"class":62,"line":306},[60,957,958],{},"            job, jobs, min_moves.value)\n",[60,960,961],{"class":62,"line":312},[60,962,177],{"emptyLinePlaceholder":176},[60,964,965],{"class":62,"line":318},[60,966,326],{},[60,968,969],{"class":62,"line":323},[60,970,177],{"emptyLinePlaceholder":176},[60,972,973],{"class":62,"line":329},[60,974,337],{},[60,976,977],{"class":62,"line":334},[60,978,343],{},[60,980,981],{"class":62,"line":340},[60,982,983],{},"        if min_moves.value == 0 or (final_moves_seq is not None\n",[60,985,986],{"class":62,"line":346},[60,987,988],{},"                                    and len(final_moves_seq) \u003C min_moves.value):\n",[60,990,991],{"class":62,"line":352},[60,992,993],{},"            min_moves.value = len(final_moves_seq)\n",[60,995,996],{"class":62,"line":358},[60,997,361],{},[60,999,1000],{"class":62,"line":364},[60,1001,1002],{},"            result['min_moves_len'] = min_moves.value\n",[60,1004,1005],{"class":62,"line":370},[60,1006,268],{},[60,1008,1009],{"class":62,"line":375},[60,1010,378],{},[60,1012,1013],{"class":62,"line":381},[60,1014,177],{"emptyLinePlaceholder":176},[60,1016,1017],{"class":62,"line":386},[60,1018,389],{},[60,1020,1021],{"class":62,"line":392},[60,1022,177],{"emptyLinePlaceholder":176},[60,1024,1025],{"class":62,"line":397},[60,1026,400],{},[60,1028,1029],{"class":62,"line":403},[60,1030,406],{},[60,1032,1033],{"class":62,"line":409},[60,1034,412],{},[60,1036,1037],{"class":62,"line":415},[60,1038,418],{},[60,1040,1041],{"class":62,"line":421},[60,1042,424],{},[60,1044,1045],{"class":62,"line":427},[60,1046,430],{},[60,1048,1049],{"class":62,"line":433},[60,1050,436],{},[60,1052,1053],{"class":62,"line":439},[60,1054,442],{},[60,1056,1057],{"class":62,"line":445},[60,1058,177],{"emptyLinePlaceholder":176},[60,1060,1061],{"class":62,"line":450},[60,1062,453],{},[60,1064,1065],{"class":62,"line":456},[60,1066,459],{},[60,1068,1069],{"class":62,"line":462},[60,1070,465],{},[60,1072,1073],{"class":62,"line":468},[60,1074,471],{},[60,1076,1077],{"class":62,"line":474},[60,1078,477],{},[60,1080,1081],{"class":62,"line":480},[60,1082,483],{},[60,1084,1085],{"class":62,"line":486},[60,1086,489],{},[60,1088,1089],{"class":62,"line":492},[60,1090,177],{"emptyLinePlaceholder":176},[60,1092,1093],{"class":62,"line":497},[60,1094,500],{},[60,1096,1097],{"class":62,"line":503},[60,1098,506],{},[60,1100,1101],{"class":62,"line":509},[60,1102,459],{},[60,1104,1105],{"class":62,"line":514},[60,1106,517],{},[60,1108,1109],{"class":62,"line":520},[60,1110,177],{"emptyLinePlaceholder":176},[60,1112,1113],{"class":62,"line":525},[60,1114,528],{},[60,1116,1117],{"class":62,"line":531},[60,1118,534],{},[60,1120,1121],{"class":62,"line":537},[60,1122,177],{"emptyLinePlaceholder":176},[60,1124,1125],{"class":62,"line":542},[60,1126,545],{},[60,1128,1129],{"class":62,"line":548},[60,1130,551],{},[60,1132,1133],{"class":62,"line":554},[60,1134,557],{},[60,1136,1137],{"class":62,"line":560},[60,1138,563],{},[60,1140,1141],{"class":62,"line":566},[60,1142,569],{},[60,1144,1145],{"class":62,"line":572},[60,1146,575],{},[60,1148,1149],{"class":62,"line":578},[60,1150,581],{},[60,1152,1153],{"class":62,"line":584},[60,1154,587],{},[60,1156,1157],{"class":62,"line":590},[60,1158,593],{},[60,1160,1161],{"class":62,"line":596},[60,1162,599],{},[60,1164,1165],{"class":62,"line":602},[60,1166,177],{"emptyLinePlaceholder":176},[60,1168,1169],{"class":62,"line":607},[60,1170,1171],{},"    min_val = mp.Value('i', 0)\n",[60,1173,1174],{"class":62,"line":613},[60,1175,610],{},[60,1177,1178],{"class":62,"line":619},[60,1179,616],{},[60,1181,1182],{"class":62,"line":625},[60,1183,1184],{},"    # os.cpu_count(). Also initialize min_moves global for each process using a \n",[60,1186,1187],{"class":62,"line":631},[60,1188,1189],{},"    # Value() shared object\n",[60,1191,1192],{"class":62,"line":637},[60,1193,1194],{},"    with mp.Pool(initializer=init_globals, initargs=(min_val,)) as pool:\n",[60,1196,1197],{"class":62,"line":643},[60,1198,634],{},[60,1200,1201],{"class":62,"line":649},[60,1202,640],{},[60,1204,1206],{"class":62,"line":1205},88,[60,1207,646],{},[60,1209,1211],{"class":62,"line":1210},89,[60,1212,652],{},[128,1214,687],{"id":1215},"the-result-1",[50,1217,1220],{"className":1218,"code":1219,"language":111},[109],"start time: 0.057702366\nentered pid: 39420: for moves: (0, 0), time: 0.14133282\npid: 39420, changed min_moves to: 18, 000001111223334444, time: 0.149716058\npid: 39420, changed min_moves to: 17, 00000111122344334, time: 0.151552746\npid: 39420, changed min_moves to: 16, 0000011112423334, time: 0.153877295\nentered pid: 39419: for moves: (0, 1), time: 0.195602278\npid: 39419, changed min_moves to: 15, 100000122344334, time: 0.203628052\npid: 39419, changed min_moves to: 14, 10000012423334, time: 0.205814167\nentered pid: 39421: for moves: (0, 3), time: 0.175389625\nentered pid: 39422: for moves: (0, 4), time: 0.230612233\npid: 39419, changed min_moves to: 13, 1004430003122, time: 2.607543404\nNo more jobs: pid: 39419, final count: 171415, time: 61.43746481\nNo more jobs: pid: 39422, final count: 210959, time: 77.746001708\nNo more jobs: pid: 39421, final count: 536682, time: 144.053795224\nNo more jobs: pid: 39420, final count: 895965, time: 210.756625277\ngot results: 211.040409704\nresult: {'min_moves': '0000011112423334', 'min_moves_len': 16, 'count': 895965, 'pid': 39420}\nresult: {'min_moves': '1004430003122', 'min_moves_len': 13, 'count': 171415, 'pid': 39419}\nresult: {'min_moves': None, 'min_moves_len': 0, 'count': 536682, 'pid': 39421}\nresult: {'min_moves': None, 'min_moves_len': 0, 'count': 210959, 'pid': 39422}\n",[57,1221,1219],{"__ignoreMap":55},[15,1223,1224],{},"Yay! We have made progress. Now it takes only 3 minutes 31 seconds to do the job (as compared to around 5 minutes 25 seconds with single process), with around 1.81 million end to end sequences.",[10,1226,1228],{"id":1227},"the-cpu-count-and-its-implications","The CPU count and its implications",[15,1230,1231,1232,1235],{},"Please note that all of the results were collected on my macbook pro having a dual core processor. So ",[57,1233,1234],{},"os.cpu_count()"," returns 4 in my case, 2 logical cores for each physical core. That is why 4 processes are getting created in the above examples. We can play around with the number of processes and see if that makes any further difference. In my case, I've found that running 2 processes gives me the best result (The value may be different for you, depending on the number of cores your workhorse has).",[15,1237,1238,1239,1241],{},"Changing one line in the ",[57,1240,134],{}," function",[15,1243,1244],{},[57,1245,1246],{},"with mp.Pool(processes=2, initializer=init_globals, initargs=(min_val,)) as pool:",[15,1248,1249],{},"We get the following results:",[50,1251,1254],{"className":1252,"code":1253,"language":111},[109],"start time: 0.103170482\nentered pid: 39540: for moves: (0, 0), time: 0.226195533\npid: 39540, changed min_moves to: 18, 000001111223334444, time: 0.238348483\npid: 39540, changed min_moves to: 17, 00000111122344334, time: 0.239980543\npid: 39540, changed min_moves to: 16, 0000011112423334, time: 0.265787826\nentered pid: 39541: for moves: (0, 1), time: 0.239561019\npid: 39541, changed min_moves to: 15, 100000122344334, time: 0.246132816\npid: 39541, changed min_moves to: 14, 10000012423334, time: 0.247991375\npid: 39541, changed min_moves to: 13, 1004430003122, time: 1.319787871\nNo more jobs: pid: 39541, final count: 171415, time: 27.815133454\nentered pid: 39541: for moves: (0, 3), time: 27.816530592\nNo more jobs: pid: 39541, final count: 533304, time: 135.0901852\nentered pid: 39541: for moves: (0, 4), time: 135.090335439\nNo more jobs: pid: 39541, final count: 207783, time: 172.9650854\nNo more jobs: pid: 39540, final count: 895570, time: 183.029733485\ngot results: 183.21633772\nresult: {'min_moves': '0000011112423334', 'min_moves_len': 16, 'count': 895570, 'pid': 39540}\nresult: {'min_moves': '1004430003122', 'min_moves_len': 13, 'count': 171415, 'pid': 39541}\nresult: {'min_moves': None, 'min_moves_len': 0, 'count': 533304, 'pid': 39541}\nresult: {'min_moves': None, 'min_moves_len': 0, 'count': 207783, 'pid': 39541}\n",[57,1255,1253],{"__ignoreMap":55},[15,1257,1258],{},"Only 2 processes with pids 39540 & 39541 were used in this case. When the process with pid 39540 was going through sequences starting with 0 column id, process with pid 39541 finished processing the sequences starting with remaining column ids (1, 3 & 4 in this case). It took nearly 3 minutes (so a saving of around 30 seconds) to finish the job, with nearly the same 1.8 million end-to-end sequences.",[15,1260,1261],{},"And we've have a winner in our midst :-)",[121,1263],{"url":1264},"https:\u002F\u002Fmedia.giphy.com\u002Fmedia\u002Fl44Q6Etd5kdSGttXa\u002Fgiphy.gif",[10,1266,1268],{"id":1267},"conclusion","Conclusion",[15,1270,1271,1272,1275,1276,1279,1280,1282],{},"That was one long article with a lot of code, and repeated optimizations. I'm sure further optimizations can be done. I also tried to distribute jobs between different processes using a shared ",[57,1273,1274],{},"queue"," object (sgain from multiprocessing), but didn't get favorable results. There is one ",[57,1277,1278],{},"Manager"," class also available in the ",[57,1281,145],{}," module, which allows us to create shared proxy objects. Using that we wouldn't need to use the initializer and initargs, and can share the object as argument like we are doing for tasks, but the documentation says that it will be slow, so I haven't covered that (of course I tried it myself :-)).",[15,1284,1285],{},"There must be other ways to solve the problem, after all, there are multiple ways to solve any problem in general, and programming in particular. It might be possible that there is a very simple trick to solve this instantly.",[15,1287,1288],{},"Do share how would you solve this problem?",[15,1290,1291],{},"Please hit me up if you have any questions, or if you find any error anywhere.",[15,1293,1294],{},"Enjoy :-)",[1296,1297,1298],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":55,"searchDepth":69,"depth":69,"links":1300},[1301,1302,1307,1312,1313],{"id":12,"depth":69,"text":13},{"id":37,"depth":69,"text":38,"children":1303},[1304,1306],{"id":130,"depth":75,"text":1305},"Modified play & find_min_moves functions",{"id":686,"depth":75,"text":687},{"id":793,"depth":69,"text":794,"children":1308},[1309,1311],{"id":808,"depth":75,"text":1310},"Modified play and find_min_moves functions",{"id":1215,"depth":75,"text":687},{"id":1227,"depth":69,"text":1228},{"id":1267,"depth":69,"text":1268},null,"\u002Fimages\u002Fposts\u002Fusing-python-multiprocessing-to-optimize-puzzle-solving\u002FiAsHu7b-l-07a58e8ed5.png","2022-11-21T12:48:43.209Z","Introduction This is the final part of the Python puzzle game series. In the first part we learnt to create a tiles puzzle game using Turtle module. And then in the second part...",false,"md","claqsasw9001c08jk2nppbp9y",{},"\u002Fusing-python-multiprocessing-to-optimize-puzzle-solving",{"title":5,"description":1317},"using-python-multiprocessing-to-optimize-puzzle-solving",[1326,54,1327,145],"tutorial","game-development","bpEJiT2z_EmTB4gPwNGqTIlkLsdepnz3T6l9kld4B0Y",1780470201970]