Made some progress:
I managed to modify GPUStats enough to start training in GPU mode. Now I'm getting:
Code: Select all
cannot import name 'get_custom_objects' from 'keras.utils'
Code: Select all
12/05/2021 14:44:41 MainProcess MainThread logger log_setup INFO Log level set to: INFO12/05/2021 14:44:41 MainProcess MainThread gpu_stats _log DEBUG Initializing GPUStats12/05/2021 14:44:41 MainProcess MainThread gpu_stats _log DEBUG Apple Silicon Detected.12/05/2021 14:44:41 MainProcess MainThread gpu_stats _log DEBUG GPU Device count: 112/05/2021 14:44:41 MainProcess MainThread gpu_stats _log DEBUG Active GPU Devices: [0]12/05/2021 14:44:41 MainProcess MainThread gpu_stats _log DEBUG GPU Handles found: 112/05/2021 14:44:41 MainProcess MainThread gpu_stats _log DEBUG [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]12/05/2021 14:44:41 MainProcess MainThread gpu_stats _log DEBUG GPU Driver: 1.012/05/2021 14:44:41 MainProcess MainThread gpu_stats _log DEBUG GPU Devices: ['Apple M1']12/05/2021 14:44:41 MainProcess MainThread gpu_stats _log DEBUG GPU VRAM: [16384]12/05/2021 14:44:41 MainProcess MainThread gpu_stats _log DEBUG Initialized GPUStats12/05/2021 14:44:41 MainProcess MainThread launcher _configure_backend DEBUG Executing: train. PID: 6528612/05/2021 14:44:41 MainProcess MainThread launcher _test_for_tf_version DEBUG Installed Tensorflow Version: 2.712/05/2021 14:44:42 MainProcess MainThread queue_manager __init__ DEBUG Initializing QueueManager12/05/2021 14:44:42 MainProcess MainThread queue_manager __init__ DEBUG Initialized QueueManager12/05/2021 14:44:42 MainProcess MainThread train __init__ DEBUG Initializing Train: (args: Namespace(batch_size=64, colab=False, configfile=None, distributed=False, exclude_gpus=None, freeze_weights=False, func=<bound method ScriptExecutor.execute_script of <lib.cli.launcher.ScriptExecutor object at 0x16bb6c0a0>>, input_a='/Users/server/Documents/_TRAINING_SETS/A/aligned', input_b='/Users/server/Documents/_TRAINING_SETS/B/aligned', iterations=1000000, load_weights=None, logfile=None, loglevel='INFO', model_dir='/Users/server/Documents/model', no_augment_color=False, no_flip=False, no_logs=False, no_warp=False, preview=False, preview_scale=100, redirect_gui=True, save_interval=250, snapshot_interval=25000, summary=False, timelapse_input_a=None, timelapse_input_b=None, timelapse_output=None, trainer='dfl-sae', warp_to_landmarks=True, write_image=False)12/05/2021 14:44:42 MainProcess MainThread train _get_images DEBUG Getting image paths12/05/2021 14:44:42 MainProcess MainThread utils get_image_paths DEBUG Scanned Folder contains 770 files12/05/2021 14:44:42 MainProcess MainThread utils get_image_paths DEBUG Returning 770 images12/05/2021 14:44:42 MainProcess MainThread train _get_images DEBUG Test file: (filename: /Users/server/Documents/_TRAINING_SETS/A/aligned/00001_0.png, metadata: {'width': 256, 'height': 256, 'itxt': {'alignments': {'x': 864, 'w': 213, 'y': -9, 'h': 294, 'landmarks_xy': [[878.9874877929688, 95.12928009033203], [870.755615234375, 126.70247650146484], [869.4014282226562, 152.7521514892578], [870.755615234375, 178.8125], [874.87158203125, 208.9781036376953], [883.1033325195312, 232.3194580078125], [891.335205078125, 250.13731384277344], [899.5670776367188, 266.6009826660156], [924.2625122070312, 285.8157043457031], [959.9088745117188, 285.8157043457031], [991.471435546875, 276.229736328125], [1016.1775512695312, 266.6009826660156], [1042.227294921875, 250.13731384277344], [1058.6910400390625, 226.83868408203125], [1071.0386962890625, 204.8621826171875], [1086.10546875, 177.4476776123047], [1097.0562744140625, 148.63621520996094], [888.5841674804688, 60.84776306152344], [895.451171875, 53.9700927734375], [910.5178833007812, 55.36699295043945], [920.1466674804688, 62.20193099975586], [931.0974731445312, 71.8306655883789], [984.6044311523438, 86.8974380493164], [1001.068115234375, 86.8974380493164], [1020.2828979492188, 91.01336669921875], [1039.465576171875, 99.24520111083984], [1055.92919921875, 114.3547134399414], [947.5612182617188, 117.10577392578125], [937.9645385742188, 140.40438842773438], [928.3784790039062, 156.86805725097656], [924.2625122070312, 173.3317413330078], [916.03076171875, 177.4476776123047], [920.1466674804688, 182.9284210205078], [928.3784790039062, 189.79541015625], [940.7262573242188, 189.79541015625], [950.312255859375, 189.79541015625], [899.5670776367188, 92.41027069091797], [905.0478515625, 89.6592025756836], [918.749755859375, 93.77511596679688], [931.0974731445312, 106.12286376953125], [916.03076171875, 106.12286376953125], [905.037109375, 100.64209747314453], [984.6044311523438, 123.94071197509766], [996.9521484375, 119.8248062133789], [1012.051025390625, 123.94071197509766], [1025.7635498046875, 134.93431091308594], [1009.2999877929688, 136.28848266601562], [992.8362426757812, 132.17254638671875], [899.5670776367188, 207.6132354736328], [903.6829223632812, 206.2590789794922], [916.03076171875, 206.2590789794922], [924.2625122070312, 210.375], [932.494384765625, 210.375], [950.312255859375, 222.72276306152344], [968.1408081054688, 232.3087615966797], [948.9580688476562, 237.78955078125], [932.494384765625, 239.1864471435547], [920.1466674804688, 236.4246826171875], [909.153076171875, 232.3087615966797], [903.6829223632812, 222.72276306152344], [900.9212036132812, 210.375], [916.03076171875, 215.8450927734375], [924.2625122070312, 218.6068572998047], [935.21337890625, 222.72276306152344], [965.4216918945312, 230.95458984375], [933.8485717773438, 224.07691955566406], [924.2625122070312, 222.72276306152344], [914.6337890625, 218.6068572998047]], 'mask': {'components': {'mask': b'x\x9c\xed\x9a\x81\r\xc20\x0c\x04\xb3YF\xf3h\x19\xad\x80\x10"-\x12r\xff?\xb5\x04\x7f\x0b\xdc\xc7NK\x13\xd3\x9a1\xc6\x18c\x8c1\xc6\x18c\x8c1\xc6\x98\xff\xa5GlOFL\xf4\x8b\xec[\x86w,\xb5\x7f\xa4\xfc\x874:}n\xf9\xeb\x12\x04\xe6\x97%\x80\xfd\xa2\x04\x84_\x92\x80\xf2\xdf\x9fX\xf6)%\xfdt\x02\xda\xbfqmP\xf8\xef\t\xe0"h\xfcx\x1bT~4\x81\xce\x8fm\x03\xfb\xed\x17\x81\xe8\xed\xb7\xffG\xfc\xa3\xd8\x8f}\x05\xe8\xfc\xd8\x0f\xb0\xce\x0f\xe9[S\xe9\xb1\xf6\xeb\xfc\xe8G`\x88\xfc\xe8\x17\xa0\xca\x0f\xeaU\r@\xdb\xaf\xf2\xe3g\x80\x90\xf8\xf1S\x90\xc6\x0f\xeb5\r\xc0\xdb\x8f\xdd\xc0\x1c\xa9>\x02R\x87`\x81\x9f\xd1\x0b\n\xc0\xb4\x1f\xbe\x03\x9b /bh?\xa7\xa7\x1b\xc0\x95\x9fo\x00}\x0f68?\xab\'\x0b\xc0\x96\x9f-@\xf55$\xaf\xa7\x1eAA\xf9\xa9\x02h\x06%\xb0\xbe\xf8"^R\xfd\x07\xc5z\xf0\x1d \x9c\x92E\xad\x1ey\tig\x84\xfd\xac^\xd7|(\x80Z\x7f\xb2\x03+\x06\xb4\xf9\x00\xf4\xe4\x89\x0b \x9f\x03\x9f\x0b\xb0p8\x9e\x08\xa0\xdfx\x13\xbdr\xf1\x89\x00\xcb:\x9f\n\xb0h\xdbg\x03\xac_\xfc\xb7\x00K\xf7\xdd\x9e(*\xfd\x94`\xec\n\x7fQ\xe5g\xfa\xab\x08W\xfd\x17\xe7\x93\xd8F\xc1\xc2\x8d1\xc6p\xdc\x00\xffVc\xad', 'affine_matrix': [[0.3200481648709836, 0.09329843938485899, -254.78927419234816], [-0.09329843938485897, 0.32004816487098364, 111.31994163088105]], 'interpolator': 2, 'stored_size': 128, 'stored_centering': 'face'}, 'extended': {'mask': b'x\x9c\xed\x9a\xdbm\xc30\x10\x04\xd5\x19K\xbb\xd2X\x1a\xf3\x11\x04\x90`\xc3\xa0v\x87"\x10\xef40\xc7[=\xf8:\x8e\x10B\x08!\x84\x10B\x08!\x84\x10\xbe\x9bV\xd5\xc7/\xbdN\xb4G\xec5\xa6\xb8TV\xc5\xf9\xe7\xf4/\xe5P\xcd\x99\x1c\xfe+P\x01]\xf5\x8f\x8e\xf8e=T\x80\xe1G2\xb0\xfc\xc3\x7f\x0f\xca+\xc0\xce\xc0\xf4\xdb\x19\xd8~3\x03\xdf\xefe\x00\xf8\xad\x0c\x1a\xe1w2@\xfcF\x06\x8c_\xcf\x80\xf2\xab\x19`~1\x03\xce\xafeP\xa0_\x89 \xfe\xef\xf67\xd0/\xe8\xd1\x170\xfe\xf8oS\x98^\xfb\x01q~\xed\x0f\xcc\xf9\xb5)\x08\xe7\x97\xf4\xdc\x03\xa8\xce\x01)\xbf:\t.\xc8\xaf\xce@)\xbf\xa8\xff/K\x00}\rT\x88__\x042~Y\xcf\x04\xe0\xac\xc2;\xe0w\xb6!\n\xf0[\xfb0\x80\xdf\xd1\x03\r\xf06\xc2\x9a\xed7w\x02m\xbf\xa7\xb7\x03p\xf7!\x9b\xe9\xb77b\xbb\xe7w\xf5f\x03\x80\xadx\xab\x01\xc0yP9~_\xbf\xff$\xc4h\x00s\x16%\xeb\xa1\xd3@\xb5\x01\xccA\xd8!\x1fERz\xf1\x1b\x00\x1e\x13\xd7^\xbd\xf2\x11b\x0f\xc9\xdb]=\x17\xbeT\x00\xad\xbf\x99\xc0\x8a\x1b\n\xf3\x05`w\x00\xb4\x02\xc0;\x10J\x01\x0bo\x87L\x14\xc0?x\'\xda\xce\xc1O\x14\xb0,\xf9\xa9\x02\x16=\xf6\xb3\x05\xac\x1f\xfc\xa7\x02\x96>wWjS\xebO\x15\xf4K\xe3\x1f\xea\xfc\x99\xf6\xd7\x84\x87.\xa3\xbd\xa1F\xdf0\xf0\x10B\x08\x1e?\xb8\xc5\x15\xd8', 'affine_matrix': [[0.3200481648709836, 0.09329843938485899, -254.78927419234816], [-0.09329843938485897, 0.32004816487098364, 111.31994163088105]], 'interpolator': 2, 'stored_size': 128, 'stored_centering': 'face'}, 'bisenet-fp_face': {'mask': b'x\x9c\xed\x9aa\x99\x84 \x10\x86m\xe05 \x82\x11\x88\xb0\x11\x88`\x04"l\x04"\x18\x81\x08F \x82\x11\xb8]\xcf}v\xef\xceue\xbeo\xf4\x0fo\x81W\x81\x19\x06\x98\xa6\xa9T*\x95J\xa5R\x91a\xac\x0f)\x9e%w)\xcf\xb8\xc3\xd5\xd6\xf9\xb8\xc8o\x1c<\x00\x9d\x9f\xf2o\x86\xf6 \xf5\x97\xf5C\xca\xff\x99.G\xd8mXQ/\xf4\xca\xee\xae\x0fk?\xfe\xe4\xaaiw\xdb\xee\x19\xa3%o\xfd\x0e{\xce\xa3\xd2"\xbc\xfc]\xed\xef\xf0*z\xb7\xd3~\x0b\x02\x05{\x1bw\xebUb\xe0Z\xa0\xcf\xd9\xb2\xf5\xdd\xde\xb9\xffa`\xfbS\x91\x9e\x1e\x83\xaePO\x1e\x00[\xaa\'g\xc1X\xee\xcf\xc4$$\xf8}j\x12\n\x12?/\x06\x8dH\x9fG\x96\xdf\xc9\xfc\xb4\x01(K}OX\xb5P\x14\xfas\xe2\xc4@\x92\xfaI\x95\x80X\xcf)\xc8;\xc0\xcf\xc8\x02\x16\xf1\x13\x8eD\x1e\xf2\xe3;!\xe8\x87wB\xd0\x0f/\x01\xd4?\x81A\x88\xfa\xd1\x01\x80\xfd\xe0\x00\xc0\xfe\x1cN\xf6c\x03\x80\xfb\xb1\x15@\xf0C\x03@\xf0C+\xa0\'\xf8\x91\x13\xa9%\xf8\x91m\x88\xe2\x07&\x80\xe2\x07V \xc5\x0fL\x00\xc7/\xdf\x869~\xf9y\xd0p\xfc\xf2\t\xe0\xf8\xe5\x87\x01\x8e_\x1e\x82$\xbf\xb8\x12e\xf9\xa5\x07b\x96_z \x8e,\x7fw\xae_\xba\t\xb2\xfc\xd2+1O\xf2K\xaf#\x18\x05\xc8\x1di\xfcA\x07\xf0\'I\xa8o\x9a\xb2\xbb\xe7w\xc8K\x90\x81\xe2\x97\xdfF9\x8a_^\x01\x19\x86\x1e\xb9\x07\x18\t~a\xf2\x9b!D\xa0|\xf57\x94\x08\xc4\x9e\xa4\x12\xec\xc7.\x83\x03\xec\x87\xf4\xcd\x05\xd5\x83w0-\xa8\x87V\xdf\x1d0\x05\xc2\xb7\xa0\xee\xdc\xdf\x07\'\x80\xf0\x12\x82L\x00\xe3\x16\x1eI\x81H\xea}`\xe4z\xceK`\x94\xea\xd1\xfb\xdf\x05q\nb=\x84&\x99\x9e\xf6\x0cie~^\'\x80(\x04\x89\xef\xf0m*\xd7S\xfb\x10l\xb1~\xe2\xf6a\x14\xbf\x04\x93\x1bq\xda\xc2B\x14{xX\xa1\xac\x10Th\x87\xf4\x05z\x95>\xb4x\xae~\x7f\x10&\xc6\xae\xb7\x82=\xf3\xef\xef\xec\xaa\x04\xa2b#l\xfc\xac\xa7\x07\xde+\xe6\xa3^\xb9\x07\xd6o\xdb\xf5{\x807\xd3\xe0\xa4\xb4\xf0_\xb0\x1b\xfaQ\xad\xfd\xf6\x85p\xae\xfe\xed\x07L\xda\xdd\xd7\x0f\xda\xd5\x0f\x08\xc7\xfc\xfcL\xff\xaf\xf7\xbe?\xaa\xf9~\xc1\xfa\xb8\x8cz\xf4\x8e\xdeq\xbb\x13c\xf5\xc3\xadR\xa9T*t\xbe\x01~^\xd1\x13', 'affine_matrix': [[0.3200481648709836, 0.09329843938485899, -254.78927419234816], [-0.09329843938485897, 0.32004816487098364, 111.31994163088105]], 'interpolator': 3, 'stored_size': 128, 'stored_centering': 'face'}}}, 'source': {'alignments_version': 2.2, 'original_filename': '00001_0.png', 'face_index': 0, 'source_filename': '00001.jpg', 'source_is_video': False, 'source_frame_dims': (1080, 1920)}}})12/05/2021 14:44:42 MainProcess MainThread train _get_images INFO Model A Directory: '/Users/server/Documents/_TRAINING_SETS/A/aligned' (770 images)12/05/2021 14:44:42 MainProcess MainThread utils get_image_paths DEBUG Scanned Folder contains 398 files12/05/2021 14:44:42 MainProcess MainThread utils get_image_paths DEBUG Returning 398 images12/05/2021 14:44:42 MainProcess MainThread train _get_images DEBUG Test file: (filename: /Users/server/Documents/_TRAINING_SETS/B/aligned/6797547230208265478_00001_0.png, metadata: {'width': 256, 'height': 256, 'itxt': {'alignments': {'x': 226, 'w': 192, 'y': 282, 'h': 280, 'landmarks_xy': [[223.76922607421875, 389.2649841308594], [221.2505340576172, 419.2222595214844], [224.9951934814453, 446.6607971191406], [228.73985290527344, 471.6474609375], [234.96849060058594, 500.3467102050781], [246.20245361328125, 522.814697265625], [261.1810607910156, 537.7932739257812], [279.9043884277344, 552.77197265625], [311.0876159667969, 565.2318725585938], [344.78955078125, 560.26123046875], [363.5128173828125, 549.0272827148438], [376.0074768066406, 537.7932739257812], [393.4700927734375, 516.5859985351562], [402.2200927734375, 489.1127014160156], [409.7093811035156, 466.644775390625], [417.19873046875, 440.4321594238281], [420.943359375, 412.9936218261719], [247.463134765625, 364.3130798339844], [261.1810607910156, 359.3103942871094], [277.42041015625, 356.82373046875], [291.1383361816406, 360.568359375], [303.6330261230469, 364.3130798339844], [358.542236328125, 371.8023986816406], [372.2628173828125, 368.0577087402344], [387.241455078125, 368.0577087402344], [402.2200927734375, 371.8023986816406], [412.193359375, 383.0363464355469], [327.326904296875, 401.7596435546875], [323.5822448730469, 420.48291015625], [323.5822448730469, 436.6875], [321.09564208984375, 451.6661376953125], [303.6330261230469, 462.900146484375], [309.86163330078125, 466.644775390625], [319.8376159667969, 470.3894958496094], [331.0715637207031, 466.644775390625], [338.5608825683594, 466.644775390625], [263.6677551269531, 390.525634765625], [274.9017028808594, 386.781005859375], [287.3936767578125, 389.2676696777344], [298.627685546875, 398.0149841308594], [287.3936767578125, 399.2409362792969], [273.6757507324219, 398.0149841308594], [353.53955078125, 402.985595703125], [368.5181579589844, 398.0149841308594], [382.236083984375, 399.2409362792969], [390.9861145019531, 405.5043029785156], [379.7521057128906, 409.2489318847656], [366.031494140625, 409.2489318847656], [281.1650390625, 500.3467102050781], [293.625, 492.857421875], [309.86163330078125, 489.1127014160156], [317.3509521484375, 489.1127014160156], [327.326904296875, 489.1127014160156], [342.3055419921875, 497.8280334472656], [353.53955078125, 507.8360290527344], [338.5608825683594, 519.0700073242188], [327.326904296875, 526.5592651367188], [313.6062927246094, 526.5592651367188], [301.1143493652344, 522.814697265625], [291.1383361816406, 515.3253784179688], [283.6490173339844, 500.3467102050781], [306.1169738769531, 500.3467102050781], [316.0929260253906, 500.3467102050781], [327.326904296875, 500.3467102050781], [349.7948913574219, 507.8360290527344], [327.326904296875, 510.3226623535156], [316.0929260253906, 511.5806579589844], [304.8589782714844, 507.8360290527344]], 'mask': {'components': {'mask': b'x\x9c\xed\x99\x01\x0e\x820\x10\x04\xf9Y\x9fvO\xeb\xd3P$F\x13\x13\xa17[.\xc6\x9d\x0f\xcc\xb1[\xc4\xb6\xcbb\x8c1\xc6\x18c\x8c1\xc6\x18c\x8c1\xe6\xdfiq\x9a\t\xee\xbe\x0e\xd1\xa5\xf6A\xb9v\x80\x8c]7@\xd2.\x1b k\xbf\xa3X\x86\xe9\xa7\xdf\xe0\xfa z\x1e@Cz\x1e\x00\xd4\xd3\x00:\xf5\xb3W \xa8~]\x1b\xd07\xaeG\x01\x08\xf4$\x80\x90\xf8\xf3\x01h\xfc\xf9W\xd0~\xfb9\xd5\xeb?\xff\x05\xb0\xff\xbf\xfd\x9a\xefO\xb5?\xaf\xd7\x14\xf0\xc3~E\x01\xe8\x0fh\xb5?~\xdfO\xf4\x8a\x02\x8a\xfdp\x0b\x1e\xd4\x0f\xf7_\xd8O\xb6?\x1b\xd4\x0f\xf5\xe5\xfe`z|\xfeP\xed\x87\x05\xd0\xe5GO \xb0\x9e\x15\xa08\x00$\xfe\xea\xf3?\x85?\x80_\xa0G\x05H\xfc\xf9\x004\xe7\xcf-\xed\x17\xdd\x82\xa4\xfd\xfc\xd7\xe7Ad\xfd\x1a}:\x00\xd9%T.\x00\xe1\x05T\xcf\xf8E\xedo\xb4\x84^z\x058\x1e\x80\xf4\xfa/\xb1\x04\xb5\xfa\xe1%(,\x7f\xa7\x0f\xe9\xc5\xe9o\xb4Z\xfd\xc8=\xe8\x14\xfdr\xbe\x03y\xf9O\xceu0M\x7f\xae\x83\x89\xfa\xe5\xb8\x83>W\x7f\xd4\xc1\xac\xa5\xf7\xce\x97\x08f?\xfcNTe\xff\x9a\xe0s\x84\x1e\x97\xd9?f\xb8\xdc\xfd6C\x99\xdb\x18c\x0c\xe1\x06\xf4\x8b\x0e\x01', 'affine_matrix': [[0.35073350169214235, 0.03703774015804282, -65.6308955358848], [-0.03703774015804283, 0.35073350169214235, -70.25819745185827]], 'interpolator': 2, 'stored_size': 128, 'stored_centering': 'face'}, 'extended': {'mask': b"x\x9c\xed\xda\x8b\x89\x031\x0c\x84aw\xa6\xd2T\x9aK\xdbc\t\xc7\x05B.k\xcd\xef\x88\x85\x99\x06>\xaf\xc6y\xd9\x19\xc3q\x1c\xc7q\x1c\xc7q\x1c\xc7q\x1c\xc7q\x9cw\x89|\x97\xa0\xa1y\xac\x05\\\xc0\xb2}fRz\x14pp\x01Y\xe4\x99\x05\xc4,\xf3\xc8\x16\x10tb\x01\xa9\xf9\xd9\xfb\xf8\xc7!\xf2\xd1\xec\xcff_\xe5\xc5\r\x90\xcd\xbe\xcck~\xe8\xbe\xb4\x01\xb2\xd9\x07x\xc5O\xc2\x176\xc0\xec\xf5\x83\xe0\x05?\x11\xbf\xbe\x01\xec\xdb\xbf\xbf_\xff\x0e\xcc\xf8\xdd\xaf\x7f\xfbw\xf5\x91\x8f\xffv\xbf\xce3\x05\xdc\xd8'\n\x90\xbe\x7fw\xfby\x7f_\xe1\x89\x02\x9a}\xf1\x04*U_<\x7f\x91}\xf5\xfcI\xf5E\xbe\xddO\x8d\x97\x8f\xdf\xba\xfd\xee\xe3O\xf17\xb8\xcck\x05\x10\xe7\xdf\x8a\xaf\xd7\xaf\x15@\xf8)\xf8\x00/\x15\x80\xf8\xf5\x010\xf7?Q\xf6\x89\xfa\x87P\x00t\x01\x98U\x9f\xe1\xcb\x03\x80\xc6_\x1d\x00v\xfdY|\x0f\x02\xaf\x7f\xa3\xc0c\xd3?\xb3>\x00p\xfag\x96}\x96_\xde\x82\xf0\x9f\x0fV\x1b\x80\xa7\x7f&z\xf9\x95\x8b\xf8-\xfc\xb8\xde\x01^\xfeo\xaeu\xb0\x8d\xbf\xd6\xc1F~|\xee`\xee\xe5?u\xb0k\xeb=\xe7\x9f\x11\xec~\xf8G\xb2k\xf6\x7f+x]\xc2\xa4\xff\xf0\xb5\xb0\x86\xaf\xdbOkh\xb3\x1d\xc7q\x1c%?\xf50\x9f\xc1", 'affine_matrix': [[0.35073350169214235, 0.03703774015804282, -65.6308955358848], [-0.03703774015804283, 0.35073350169214235, -70.25819745185827]], 'interpolator': 2, 'stored_size': 128, 'stored_centering': 'face'}, 'bisenet-fp_face': {'mask': b"x\x9c\xed\x9b\r\x95\xab0\x10FqP\tHX\tH@\x02\x12\x90\x80\x04$DB%\x8c\x04$ \x01\ty-\xdd\xb3\xfb\x1e-]2\xf7\x03\xce;\x9b\xcf\xc0\x85\xc9\xfc%\x99\x14EVVVVVV\xd6\x8f*[\x9b\xd5w\x1f'\xd0\xab\x10\xbfemy,\xfc\x1a\x97\x9a\xae\xed\xe5 z\x19\x9e\xe8\x8fO\xe8\x0fY\x88zz\x8d\x9f?\xc1v\xff\x86f\x9d\xfe\xd0\xd0V\xfb\xad\xc4e\xc5\xf6\x0b\x8d\xcdN\xfcg\xc7[\x91U{\xe0\xdb\xad\xf8\xbb\r\xf4\x9eP'\xe0o\xce(_\x841\x89\x1fc\xd0\xe2\x9bD|\x8c\xbd\x12_\xbe\t\xfc5)\xbd0\xa4\xe3\xe3\xa4\xfb\x80\x8b\x03\x7f\x93,\n\x1a\x1f\x7fP%\xc3\xc1\xc7\x8f\xa6\xc1WN\xbc\xca\x07\x83\x9b\xafY\x01s\xf3c\xa7\xe0{\x97\xff\xa6I\xc1\xf7\xe351H\xf8\x8a:@\xf8Q\xd0\x1e#\xbe\xa0\x0e!\xbe \x07!~<\x9b\x8f\x1d\xe0\x83\xf1q\x0e\xae\x18\x1f\xb7\x82)\x9d\xef\x0b\xe1\x14\xdc1\xfe\x95\xf2\x03\xe3\xe3\x004\xc6\xc7\x01\xe8\xe8}\xffQ\xcd\xf0%\xc4S\x07lN\xe6w\x94\x0f\x03\xc0(\x1f\x06\x00u\xbf8 <\xcc\xfew!>\xcc\xbe\x98\xdfs>\xaa\xc0\xc6\xf9\xa8\x02c\xf7\x83\x15\x98\xe3Q\x02\xaa\x04|\x92\x00\x14\xfc\xf1d>\t@\t\x1f\xec\xc2%|\x10\x80\xbf\x9d/(?,\x01*\xf8\xa4\x00\x08\xf2/*\x80\xc6\xf1\xa8\x01\t\x9c\x8f\x1a@A\xff\x81\x1a\xe0\x8a\xf3\xd9\x0e\x98\xf3\xd9\t\x80a>;\x84\xc5\r )\xbf\x85\xc0\x01\xe1\xfe\xa3\xa2|\xb8\xffs^\xbe|\x0b\xee\xbf\xc9\xf1\xf7,z\x07\x10\x18\x9em\xff\x8a\xf3\xcf\xbf`\x0b@\x97\x9ff@~\x03a\x88\x8f\xf1\xec\x04Fp\xfe\x8e\x1c@q\x036\x02\xbe\xe2\x02*\xf8\xf1\xb0\xf8<\xd4\xf8\xf9\x921\x04P\x024w\xf0\xc1\x8b\x97\x98\x1f\xd4`\xd5\x14\x889\xf9\xaa\x11\x08\xa7\x01\xf0\xdd\xc7\x97\xc6S\x7f\xdf\x19\x82\xb8\xf4\xff%\x8f\x01\x94SXi\x03`\xb3&\xe90\xa0\x9d\xfa\xfb\x8e\x10\x10M\xdf|\xa9K\xc3O\xf2\xc1\xdc\xb4F\x9c\xf7}K%\xf5!j\xeb\xdf\x95\xd2\x89\xef2\x04\x1a6\xe3%\x83GO\xbat\x1b\xf1\xb2\xd1\xb7\xa5\x9aM\xe7q\xbb\xe1\x8b\xa2\xdc0\x07\xbc#\xbex?\x05>\xeb\xba\xf38|io\xf1\xd2\xc9\xd7\xd7\xaa\xc7u\xdb\x1f\xf3$\xa2ym\x83\xf1\xb0\xa7\x08\xb7\x82\xd4/\xac0\xb4G?\x07\xa9;\xfb,\n\x16\x0e~\x08\x92\x95\x95\x95\x95\xf5\xdf\xeb\x0fP\x00\xc6\xa3", 'affine_matrix': [[0.35073350169214235, 0.03703774015804282, -65.6308955358848], [-0.03703774015804283, 0.35073350169214235, -70.25819745185827]], 'interpolator': 3, 'stored_size': 128, 'stored_centering': 'face'}}}, 'source': {'alignments_version': 2.2, 'original_filename': '6797547230208265478_00001_0.png', 'face_index': 0, 'source_filename': '6797547230208265478_00001.jpg', 'source_is_video': False, 'source_frame_dims': (1280, 720)}}})12/05/2021 14:44:42 MainProcess MainThread train _get_images INFO Model B Directory: '/Users/server/Documents/_TRAINING_SETS/B/aligned' (398 images)12/05/2021 14:44:42 MainProcess MainThread train _get_images DEBUG Got image paths: [('a', '770 images'), ('b', '398 images')]12/05/2021 14:44:42 MainProcess MainThread train __init__ DEBUG Initialized Train12/05/2021 14:44:42 MainProcess MainThread train process DEBUG Starting Training Process12/05/2021 14:44:42 MainProcess MainThread train process INFO Training data directory: /Users/server/Documents/model12/05/2021 14:44:42 MainProcess MainThread train _start_thread DEBUG Launching Trainer thread12/05/2021 14:44:42 MainProcess MainThread multithreading __init__ DEBUG Initializing MultiThread: (target: '_training', thread_count: 1)12/05/2021 14:44:42 MainProcess MainThread multithreading __init__ DEBUG Initialized MultiThread: '_training'12/05/2021 14:44:42 MainProcess MainThread multithreading start DEBUG Starting thread(s): '_training'12/05/2021 14:44:42 MainProcess MainThread multithreading start DEBUG Starting thread 1 of 1: '_training_0'12/05/2021 14:44:42 MainProcess MainThread multithreading start DEBUG Started all threads '_training': 112/05/2021 14:44:42 MainProcess MainThread train _start_thread DEBUG Launched Trainer thread12/05/2021 14:44:42 MainProcess MainThread train _monitor DEBUG Launching Monitor12/05/2021 14:44:42 MainProcess MainThread train _monitor INFO ===================================================12/05/2021 14:44:42 MainProcess MainThread train _monitor INFO Starting12/05/2021 14:44:42 MainProcess MainThread train _monitor INFO Press 'Stop' to save and quit12/05/2021 14:44:42 MainProcess MainThread train _monitor INFO ===================================================12/05/2021 14:44:43 MainProcess _training_0 train _training DEBUG Commencing Training12/05/2021 14:44:43 MainProcess _training_0 train _training INFO Loading data, this may take a while...12/05/2021 14:44:43 MainProcess _training_0 train _load_model DEBUG Loading Model12/05/2021 14:44:43 MainProcess _training_0 utils get_folder DEBUG Requested path: '/Users/server/Documents/model'12/05/2021 14:44:43 MainProcess _training_0 utils get_folder DEBUG Returning: '/Users/server/Documents/model'12/05/2021 14:44:43 MainProcess _training_0 plugin_loader _import INFO Loading Model from Dfl_Sae plugin...12/05/2021 14:44:43 MainProcess _training_0 multithreading run DEBUG Error in thread (_training_0): cannot import name 'get_custom_objects' from 'keras.utils' (/Users/server/miniforge3/envs/faceswap-env/lib/python3.8/site-packages/keras/utils/__init__.py)12/05/2021 14:44:44 MainProcess MainThread train _monitor DEBUG Thread error detected12/05/2021 14:44:44 MainProcess MainThread train _monitor DEBUG Closed Monitor12/05/2021 14:44:44 MainProcess MainThread train _end_thread DEBUG Ending Training thread12/05/2021 14:44:44 MainProcess MainThread train _end_thread CRITICAL Error caught! Exiting...12/05/2021 14:44:44 MainProcess MainThread multithreading join DEBUG Joining Threads: '_training'12/05/2021 14:44:44 MainProcess MainThread multithreading join DEBUG Joining Thread: '_training_0'12/05/2021 14:44:44 MainProcess MainThread multithreading join ERROR Caught exception in thread: '_training_0'Traceback (most recent call last): File "/Users/server/Documents/faceswap/lib/cli/launcher.py", line 182, in execute_script process.process() File "/Users/server/Documents/faceswap/scripts/train.py", line 190, in process self._end_thread(thread, err) File "/Users/server/Documents/faceswap/scripts/train.py", line 230, in _end_thread thread.join() File "/Users/server/Documents/faceswap/lib/multithreading.py", line 121, in join raise thread.err[1].with_traceback(thread.err[2]) File "/Users/server/Documents/faceswap/lib/multithreading.py", line 37, in run self._target(*self._args, **self._kwargs) File "/Users/server/Documents/faceswap/scripts/train.py", line 252, in _training raise err File "/Users/server/Documents/faceswap/scripts/train.py", line 240, in _training model = self._load_model() File "/Users/server/Documents/faceswap/scripts/train.py", line 264, in _load_model model = PluginLoader.get_model(self._args.trainer)( File "/Users/server/Documents/faceswap/plugins/plugin_loader.py", line 97, in get_model return PluginLoader._import("train.model", name, disable_logging) File "/Users/server/Documents/faceswap/plugins/plugin_loader.py", line 163, in _import module = import_module(mod) File "/Users/server/miniforge3/envs/faceswap-env/lib/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 843, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/Users/server/Documents/faceswap/plugins/train/model/dfl_sae.py", line 10, in <module> from lib.model.nn_blocks import Conv2DOutput, Conv2DBlock, ResidualBlock, UpscaleBlock File "/Users/server/Documents/faceswap/lib/model/__init__.py", line 6, in <module> from .normalization import * File "/Users/server/Documents/faceswap/lib/model/normalization/__init__.py", line 5, in <module> from .normalization_common import AdaInstanceNormalization File "/Users/server/Documents/faceswap/lib/model/normalization/normalization_common.py", line 10, in <module> from keras.utils import get_custom_objectsImportError: cannot import name 'get_custom_objects' from 'keras.utils' (/Users/server/miniforge3/envs/faceswap-env/lib/python3.8/site-packages/keras/utils/__init__.py)============ System Information ============encoding: UTF-8git_branch: mastergit_commits: fb0afa2 Revert "Trick into doing GPU training using Nvidia backend setting". c212dd4 Add Apple backend. 19084be Get shared RAM info using psutil. 5187df9 Trick into doing GPU training using Nvidia backend setting. 5db622e Update install guidegpu_cuda: No global version found. Check Conda packages for Conda Cudagpu_cudnn: No global version found. Check Conda packages for Conda cuDNNgpu_devices: GPU_0: Apple M1gpu_devices_active: GPU_0gpu_driver: 1.0gpu_vram: GPU_0: 16384MBos_machine: arm64os_platform: macOS-12.0.1-arm64-arm-64bitos_release: 21.1.0py_command: /Users/server/Documents/faceswap/faceswap.py train -A /Users/server/Documents/_TRAINING_SETS/A/aligned -B /Users/server/Documents/_TRAINING_SETS/B/aligned -m /Users/server/Documents/model -t dfl-sae -bs 64 -it 1000000 -s 250 -ss 25000 -ps 100 -wl -L INFO -guipy_conda_version: conda 4.10.3py_implementation: CPythonpy_version: 3.8.12py_virtual_env: Truesys_cores: 10sys_processor: armsys_ram: Total: 16384MB, Available: 10665MB, Used: 4360MB, Free: 8333MB=============== Pip Packages ===============absl-py @ file:///home/conda/feedstock_root/build_artifacts/absl-py_1606234718434/workaiohttp @ file:///Users/runner/miniforge3/conda-bld/aiohttp_1637087375815/workaiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1636093929600/workastunparse @ file:///home/conda/feedstock_root/build_artifacts/astunparse_1610696312422/workasync-timeout @ file:///home/conda/feedstock_root/build_artifacts/async-timeout_1637092647930/workattrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1620387926260/workblinker==1.4brotlipy @ file:///Users/runner/miniforge3/conda-bld/brotlipy_1636012322014/workcached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/workcachetools @ file:///home/conda/feedstock_root/build_artifacts/cachetools_1633010882559/workcertifi==2021.10.8cffi @ file:///Users/runner/miniforge3/conda-bld/cffi_1636046166270/workcharset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1637858084653/workclick @ file:///Users/runner/miniforge3/conda-bld/click_1635822678136/workcolorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1602866480661/workcryptography @ file:///Users/runner/miniforge3/conda-bld/cryptography_1636040761681/workcycler==0.11.0dataclasses @ file:///home/conda/feedstock_root/build_artifacts/dataclasses_1628958434797/workffmpy==0.2.3flatbuffers==2.0frozenlist @ file:///Users/runner/miniforge3/conda-bld/frozenlist_1636504246445/workgast @ file:///home/conda/feedstock_root/build_artifacts/gast_1596839682936/workgoogle-auth @ file:///home/conda/feedstock_root/build_artifacts/google-auth_1629296548061/workgoogle-auth-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/google-auth-oauthlib_1630497468950/workgoogle-pasta==0.2.0grpcio @ file:///Users/runner/miniforge3/conda-bld/grpcio_1637189242005/workh5py @ file:///Users/runner/miniforge3/conda-bld/h5py_1609497512060/workidna @ file:///home/conda/feedstock_root/build_artifacts/idna_1609836280497/workimageio @ file:///home/conda/feedstock_root/build_artifacts/imageio_1638534663447/workimageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1629987409325/workimportlib-metadata @ file:///Users/runner/miniforge3/conda-bld/importlib-metadata_1636431683784/workjoblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1633637554808/workkeras @ file:///home/conda/feedstock_root/build_artifacts/keras_1637872120405/work/keras-2.7.0-py2.py3-none-any.whlKeras-Preprocessing @ file:///home/conda/feedstock_root/build_artifacts/keras-preprocessing_1610713559828/workkiwisolver==1.3.2libclang==12.0.0Markdown @ file:///home/conda/feedstock_root/build_artifacts/markdown_1637220118004/workmatplotlib==3.2.2multidict @ file:///Users/runner/miniforge3/conda-bld/multidict_1636019232776/worknumpy==1.21.4nvidia-ml-py==11.495.46oauthlib @ file:///home/conda/feedstock_root/build_artifacts/oauthlib_1622563202229/workolefile @ file:///home/conda/feedstock_root/build_artifacts/olefile_1602866521163/workopencv-python==4.5.4.58opt-einsum @ file:///home/conda/feedstock_root/build_artifacts/opt_einsum_1617859230218/workPillow @ file:///Users/runner/miniforge3/conda-bld/pillow_1636559170621/workprotobuf==3.19.1psutil @ file:///Users/runner/miniforge3/conda-bld/psutil_1635822822120/workpyasn1==0.4.8pyasn1-modules==0.2.7pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/workPyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1634405536383/workpyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1633192417276/workpyparsing==3.0.6PySocks @ file:///Users/runner/miniforge3/conda-bld/pysocks_1635862741516/workpython-dateutil==2.8.2pyu2f @ file:///home/conda/feedstock_root/build_artifacts/pyu2f_1604248910016/workrequests @ file:///home/conda/feedstock_root/build_artifacts/requests_1637771257551/workrequests-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/requests-oauthlib_1595492159598/workrsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1637781155505/workscikit-learn @ file:///Users/runner/miniforge3/conda-bld/scikit-learn_1636784348537/workscipy @ file:///Users/runner/miniforge3/conda-bld/scipy_1637806815742/worksix @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/worktensorboard @ file:///home/conda/feedstock_root/build_artifacts/tensorboard_1629677129676/work/tensorboard-2.6.0-py3-none-any.whltensorboard-data-server @ file:///Users/runner/miniforge3/conda-bld/tensorboard-data-server_1636046148099/work/tensorboard_data_server-0.6.0-py3-none-macosx_11_0_arm64.whltensorboard-plugin-wit @ file:///home/conda/feedstock_root/build_artifacts/tensorboard-plugin-wit_1611075653546/work/tensorboard_plugin_wit-1.8.0-py3-none-any.whltensorflow-estimator==2.7.0tensorflow-macos==2.7.0tensorflow-metal==0.3.0termcolor==1.1.0threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1633102299089/worktqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1632160078689/worktyping-extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1602702424206/workurllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1632350318291/workWerkzeug @ file:///home/conda/feedstock_root/build_artifacts/werkzeug_1621518206714/workwrapt @ file:///Users/runner/miniforge3/conda-bld/wrapt_1624971819058/workyarl @ file:///Users/runner/miniforge3/conda-bld/yarl_1636047129772/workzipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1633302054558/work============== Conda Packages ==============# packages in environment at /Users/server/miniforge3/envs/faceswap-env:## Name Version Build Channelabsl-py 0.10.0 pyhd8ed1ab_1 conda-forgeaiohttp 3.8.1 py38hea4295b_0 conda-forgeaiosignal 1.2.0 pyhd8ed1ab_0 conda-forgeaom 3.2.0 hc470f4d_2 conda-forgeastunparse 1.6.3 pyhd8ed1ab_0 conda-forgeasync-timeout 4.0.1 pyhd8ed1ab_0 conda-forgeattrs 21.2.0 pyhd8ed1ab_0 conda-forgeblinker 1.4 py_1 conda-forgebrotlipy 0.7.0 py38hea4295b_1003 conda-forgebzip2 1.0.8 h3422bc3_4 conda-forgec-ares 1.18.1 h3422bc3_0 conda-forgeca-certificates 2021.10.8 h4653dfc_0 conda-forgecached-property 1.5.2 hd8ed1ab_1 conda-forgecached_property 1.5.2 pyha770c72_1 conda-forgecachetools 4.2.4 pyhd8ed1ab_0 conda-forgecertifi 2021.10.8 py38h10201cd_1 conda-forgecffi 1.15.0 py38hc67bbb8_0 conda-forgecharset-normalizer 2.0.8 pyhd8ed1ab_0 conda-forgeclick 8.0.3 py38h10201cd_1 conda-forgecolorama 0.4.4 pyh9f0ad1d_0 conda-forgecryptography 35.0.0 py38h10d4710_2 conda-forgedataclasses 0.8 pyhc8e2a94_3 conda-forgeffmpeg 4.4.1 hdbd4ad8_0 conda-forgeflatbuffers 2.0 pypi_0 pypifreetype 2.10.4 h17b34a0_1 conda-forgefrozenlist 1.2.0 py38hea4295b_1 conda-forgegast 0.4.0 pyh9f0ad1d_0 conda-forgegettext 0.19.8.1 h049c9fb_1008 conda-forgegmp 6.2.1 h9f76cd9_0 conda-forgegnutls 3.6.13 h706517b_1 conda-forgegoogle-auth 1.35.0 pyh6c4a22f_0 conda-forgegoogle-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forgegoogle-pasta 0.2.0 pyh8c360ce_0 conda-forgegrpcio 1.42.0 py38h69ee544_0 conda-forgeh5py 3.1.0 nompi_py38h032b01a_100 conda-forgehdf5 1.10.6 nompi_h0fc092c_1114 conda-forgeicu 69.1 hbdafb3b_0 conda-forgeidna 3.1 pyhd3deb0d_0 conda-forgeimageio 2.13.1 pyhd8ed1ab_1 conda-forgeimageio-ffmpeg 0.4.5 pyhd8ed1ab_0 conda-forgeimportlib-metadata 4.8.2 py38h10201cd_0 conda-forgejbig 2.1 h3422bc3_2003 conda-forgejoblib 1.1.0 pyhd8ed1ab_0 conda-forgejpeg 9d h27ca646_0 conda-forgekeras 2.7.0 pyhd8ed1ab_0 conda-forgekeras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forgekrb5 1.19.2 hd92b7a7_3 conda-forgelame 3.100 h27ca646_1001 conda-forgelcms2 2.12 had6a04f_0 conda-forgelerc 3.0 hbdafb3b_0 conda-forgelibblas 3.9.0 12_osxarm64_openblas conda-forgelibcblas 3.9.0 12_osxarm64_openblas conda-forgelibclang 12.0.0 pypi_0 pypilibcurl 7.80.0 h8fe1914_1 conda-forgelibcxx 12.0.1 h168391b_0 conda-forgelibdeflate 1.8 h3422bc3_0 conda-forgelibedit 3.1.20191231 hc8eb9b7_2 conda-forgelibev 4.33 h642e427_1 conda-forgelibffi 3.4.2 h3422bc3_5 conda-forgelibgfortran 5.0.0.dev0 11_0_1_hf114ba7_23 conda-forgelibgfortran5 11.0.1.dev0 hf114ba7_23 conda-forgelibiconv 1.16 h642e427_0 conda-forgeliblapack 3.9.0 12_osxarm64_openblas conda-forgelibllvm11 11.1.0 h93073aa_2 conda-forgelibnghttp2 1.43.0 he4cd7f6_1 conda-forgelibopenblas 0.3.18 openmp_h5dd58f0_0 conda-forgelibpng 1.6.37 hf7e6567_2 conda-forgelibprotobuf 3.19.1 hccf11d3_0 conda-forgelibssh2 1.10.0 hb80f160_2 conda-forgelibtiff 4.3.0 h74060c4_2 conda-forgelibvpx 1.11.0 hc470f4d_3 conda-forgelibwebp-base 1.2.1 h3422bc3_0 conda-forgelibxml2 2.9.12 hedbfbf4_1 conda-forgelibzlib 1.2.11 hee7b306_1013 conda-forgellvm-openmp 12.0.1 hf3c4609_1 conda-forgelz4-c 1.9.3 hbdafb3b_1 conda-forgemarkdown 3.3.6 pyhd8ed1ab_0 conda-forgemultidict 5.2.0 py38hea4295b_1 conda-forgencurses 6.2 h9aa5885_4 conda-forgenettle 3.6 hc6a1b29_0 conda-forgenumpy 1.19.5 py38hbf7bb01_2 conda-forgeoauthlib 3.1.1 pyhd8ed1ab_0 conda-forgeolefile 0.46 pyh9f0ad1d_1 conda-forgeopenh264 2.1.1 habe5f53_0 conda-forgeopenjpeg 2.4.0 h062765e_1 conda-forgeopenssl 1.1.1l h3422bc3_0 conda-forgeopt_einsum 3.3.0 pyhd8ed1ab_1 conda-forgepillow 8.4.0 py38h02acf36_0 conda-forgepip 21.3.1 pyhd8ed1ab_0 conda-forgeprotobuf 3.19.1 py38h6f2b01f_1 conda-forgepsutil 5.8.0 py38hea4295b_2 conda-forgepyasn1 0.4.8 py_0 conda-forgepyasn1-modules 0.2.7 py_0 conda-forgepycparser 2.21 pyhd8ed1ab_0 conda-forgepyjwt 2.3.0 pyhd8ed1ab_0 conda-forgepyopenssl 21.0.0 pyhd8ed1ab_0 conda-forgepysocks 1.7.1 py38h10201cd_4 conda-forgepython 3.8.12 hab31e5c_2_cpython conda-forgepython_abi 3.8 2_cp38 conda-forgepyu2f 0.1.5 pyhd8ed1ab_0 conda-forgereadline 8.1 hedafd6a_0 conda-forgerequests 2.26.0 pyhd8ed1ab_1 conda-forgerequests-oauthlib 1.3.0 pyh9f0ad1d_0 conda-forgersa 4.8 pyhd8ed1ab_0 conda-forgescikit-learn 1.0.1 py38h2cd4032_2 conda-forgescipy 1.7.3 py38hd0c9ec0_0 conda-forgesetuptools 59.4.0 py38h10201cd_0 conda-forgesix 1.15.0 pyh9f0ad1d_0 conda-forgesqlite 3.37.0 h72a2b83_0 conda-forgesvt-av1 0.8.7 hc470f4d_1 conda-forgetensorboard 2.6.0 pyhd8ed1ab_1 conda-forgetensorboard-data-server 0.6.0 py38h10d4710_1 conda-forgetensorboard-plugin-wit 1.8.0 pyh44b312d_0 conda-forgetensorflow-deps 2.7.0 0 appletensorflow-estimator 2.7.0 pypi_0 pypitensorflow-macos 2.7.0 pypi_0 pypitensorflow-metal 0.3.0 pypi_0 pypitermcolor 1.1.0 py_2 conda-forgethreadpoolctl 3.0.0 pyh8a188c0_0 conda-forgetk 8.6.11 he1e0b03_1 conda-forgetqdm 4.62.3 pyhd8ed1ab_0 conda-forgetyping-extensions 3.7.4.3 0 conda-forgetyping_extensions 3.7.4.3 py_0 conda-forgeurllib3 1.26.7 pyhd8ed1ab_0 conda-forgewerkzeug 2.0.1 pyhd8ed1ab_0 conda-forgewheel 0.35.1 pyh9f0ad1d_0 conda-forgewrapt 1.12.1 py38hea4295b_3 conda-forgex264 1!161.3030 h3422bc3_1 conda-forgex265 3.5 h666519e_1 conda-forgexz 5.2.5 h642e427_1 conda-forgeyarl 1.7.2 py38hea4295b_1 conda-forgezipp 3.6.0 pyhd8ed1ab_0 conda-forgezlib 1.2.11 hee7b306_1013 conda-forgezstd 1.5.0 h861e0a7_0 conda-forge================= Configs ==================--------- convert.ini ---------[color.color_transfer]clip: Truepreserve_paper: True[color.match_hist]threshold: 99.0[color.manual_balance]colorspace: HSVbalance_1: 0.0balance_2: 0.0balance_3: 0.0contrast: 0.0brightness: 0.0[writer.pillow]format: pngdraw_transparent: Falseoptimize: Falsegif_interlace: Truejpg_quality: 75png_compress_level: 3tif_compression: tiff_deflate[writer.ffmpeg]container: mp4codec: libx264crf: 23preset: mediumtune: noneprofile: autolevel: autoskip_mux: False[writer.gif]fps: 25loop: 0palettesize: 256subrectangles: False[writer.opencv]format: pngdraw_transparent: Falsejpg_quality: 75png_compress_level: 3[mask.mask_blend]type: normalizedkernel_size: 3passes: 4threshold: 4erosion: 0.0[mask.box_blend]type: gaussiandistance: 11.0radius: 5.0passes: 1[scaling.sharpen]method: noneamount: 150radius: 0.3threshold: 5.0--------- gui.ini ---------[global]fullscreen: Falsetab: extractoptions_panel_width: 30console_panel_height: 20icon_size: 14font: defaultfont_size: 9autosave_last_session: prompttimeout: 120auto_load_model_stats: True--------- .faceswap ---------backend: apple--------- extract.ini ---------[global]allow_growth: False[detect.mtcnn]minsize: 20scalefactor: 0.709batch-size: 8threshold_1: 0.6threshold_2: 0.7threshold_3: 0.7[detect.cv2_dnn]confidence: 50[detect.s3fd]confidence: 70batch-size: 4[align.fan]batch-size: 12[mask.unet_dfl]batch-size: 8[mask.vgg_obstructed]batch-size: 2[mask.vgg_clear]batch-size: 6[mask.bisenet_fp]batch-size: 8include_ears: Falseinclude_hair: Falseinclude_glasses: True--------- train.ini ---------[global]centering: facecoverage: 87.5icnr_init: Falseconv_aware_init: Falseoptimizer: adamlearning_rate: 5e-05epsilon_exponent: -7reflect_padding: Falseallow_growth: Falsemixed_precision: Falsenan_protection: Trueconvert_batchsize: 16[global.loss]loss_function: ssimmask_loss_function: msel2_reg_term: 100eye_multiplier: 3mouth_multiplier: 2penalized_mask_loss: Truemask_type: extendedmask_blur_kernel: 3mask_threshold: 4learn_mask: False[model.phaze_a]output_size: 128shared_fc: noneenable_gblock: Truesplit_fc: Truesplit_gblock: Falsesplit_decoders: Falseenc_architecture: fs_originalenc_scaling: 40enc_load_weights: Truebottleneck_type: densebottleneck_norm: nonebottleneck_size: 1024bottleneck_in_encoder: Truefc_depth: 1fc_min_filters: 1024fc_max_filters: 1024fc_dimensions: 4fc_filter_slope: -0.5fc_dropout: 0.0fc_upsampler: upsample2dfc_upsamples: 1fc_upsample_filters: 512fc_gblock_depth: 3fc_gblock_min_nodes: 512fc_gblock_max_nodes: 512fc_gblock_filter_slope: -0.5fc_gblock_dropout: 0.0dec_upscale_method: subpixeldec_norm: nonedec_min_filters: 64dec_max_filters: 512dec_filter_slope: -0.45dec_res_blocks: 1dec_output_kernel: 5dec_gaussian: Truedec_skip_last_residual: Truefreeze_layers: keras_encoderload_layers: encoderfs_original_depth: 4fs_original_min_filters: 128fs_original_max_filters: 1024mobilenet_width: 1.0mobilenet_depth: 1mobilenet_dropout: 0.001[model.realface]input_size: 64output_size: 128dense_nodes: 1536complexity_encoder: 128complexity_decoder: 512[model.dfl_sae]input_size: 128clipnorm: Truearchitecture: dfautoencoder_dims: 0encoder_dims: 42decoder_dims: 21multiscale_decoder: False[model.unbalanced]input_size: 128lowmem: Falseclipnorm: Truenodes: 1024complexity_encoder: 128complexity_decoder_a: 384complexity_decoder_b: 512[model.dlight]features: bestdetails: goodoutput_size: 256[model.villain]lowmem: False[model.dfaker]output_size: 128[model.original]lowmem: False[model.dfl_h128]lowmem: False[trainer.original]preview_images: 14zoom_amount: 5rotation_range: 10shift_range: 5flip_chance: 50color_lightness: 30color_ab: 8color_clahe_chance: 50color_clahe_max_size: 4
Not sure if this is related to a change in Tensorflow version 2.7? I tried changing the line
Code: Select all
from keras.utils import get_custom_objects
Code: Select all
from tensorflow.keras.utils import get_custom_objects
and a couple of other places. Now training actually runs, the training preview appears and I'm seeing GPU usage in the Activity Monitor! But after a few seconds, activity drops and training seems to be stuck.