{ "cells": [ { "cell_type": "markdown", "id": "05fc7b5c", "metadata": {}, "source": [ "Copyright (c) MONAI Consortium \n", "Licensed under the Apache License, Version 2.0 (the \"License\"); \n", "you may not use this file except in compliance with the License. \n", "You may obtain a copy of the License at \n", "    http://www.apache.org/licenses/LICENSE-2.0 \n", "Unless required by applicable law or agreed to in writing, software \n", "distributed under the License is distributed on an \"AS IS\" BASIS, \n", "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. \n", "See the License for the specific language governing permissions and \n", "limitations under the License." ] }, { "cell_type": "markdown", "id": "777b7dcb", "metadata": {}, "source": [ "# Training a 3D ControlNet for Generating 3D Images Based on Input Masks \n", "\n", "![Generated image examples and input mask](https://developer.download.nvidia.com/assets/Clara/Images/monai_maisi_ct_generative_example_synthetic_data.png)\n", "\n", "In this notebook, we detail the procedure for training a 3D ControlNet to generate high-dimensional 3D medical images. Due to the potential for out-of-memory issues on most GPUs when generating large images (e.g., those with dimensions of 512 x 512 x 512 or greater), we have structured the training process into two primary steps: 1) preparing training data, 2) training config preparation, and 3) launch training of 3D ControlNet. The subsequent sections will demonstrate the entire process using a simulated dataset. We also provide the real preprocessed dataset used in the finetuning config `environment_maisi_controlnet_train.json`. More instructions about how to preprocess real data can be found in the [README](./data/README.md) in `data` folder.\n", "\n", "`[Release Note (March 2025)]:` We are excited to announce the new MAISI Version `'maisi3d-rflow'`. Compared with the previous version `'maisi3d-ddpm'`, it accelerated latent diffusion model inference by 33x. Please see the detailed difference in the following section." ] }, { "cell_type": "markdown", "id": "c9ecfb90", "metadata": {}, "source": [ "## Setup environment" ] }, { "cell_type": "code", "execution_count": 1, "id": "58cbde9b", "metadata": {}, "outputs": [], "source": [ "!python -c \"import monai\" || pip install -q \"monai-weekly[pillow, tqdm]\"" ] }, { "cell_type": "markdown", "id": "d655b95c", "metadata": {}, "source": [ "## Setup imports" ] }, { "cell_type": "code", "execution_count": 2, "id": "e3bf0346", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "MONAI version: 1.4.1rc1+32.g34f37973\n", "Numpy version: 1.26.4\n", "Pytorch version: 2.5.0+cu124\n", "MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False\n", "MONAI rev id: 34f379735c5e18e7f809453eb1b3606c225c788b\n", "MONAI __file__: /localhome//.local/lib/python3.10/site-packages/monai/__init__.py\n", "\n", "Optional dependencies:\n", "Pytorch Ignite version: 0.4.11\n", "ITK version: 5.4.0\n", "Nibabel version: 5.3.2\n", "scikit-image version: 0.24.0\n", "scipy version: 1.14.1\n", "Pillow version: 11.0.0\n", "Tensorboard version: 2.18.0\n", "gdown version: 5.2.0\n", "TorchVision version: 0.20.0+cu124\n", "tqdm version: 4.66.5\n", "lmdb version: 1.5.1\n", "psutil version: 6.1.0\n", "pandas version: 2.2.3\n", "einops version: 0.8.0\n", "transformers version: 4.40.2\n", "mlflow version: 2.17.1\n", "pynrrd version: 1.0.0\n", "clearml version: 1.16.5rc2\n", "\n", "For details about installing the optional dependencies, please visit:\n", " https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies\n", "\n" ] } ], "source": [ "import copy\n", "import json\n", "import os\n", "import subprocess\n", "\n", "import nibabel as nib\n", "import numpy as np\n", "from monai.config import print_config\n", "from monai.data import create_test_image_3d\n", "from scripts.diff_model_setting import setup_logging\n", "from IPython.display import Image, display\n", "\n", "print_config()\n", "\n", "logger = setup_logging(\"notebook\")" ] }, { "cell_type": "markdown", "id": "b16b92a4-2039-42b7-bf77-68851d25701b", "metadata": {}, "source": [ "## Set up the MAISI version\n", "\n", "Choose between `'maisi3d-ddpm'` and `'maisi3d-rflow'`. The differences are:\n", "- The maisi version `'maisi3d-ddpm'` uses basic noise scheduler DDPM. `'maisi3d-rflow'` uses Rectified Flow scheduler, can be 33 times faster during inference.\n", "- The maisi version `'maisi3d-ddpm'` requires training images to be labeled with body regions (`\"top_region_index\"` and `\"bottom_region_index\"`), while `'maisi3d-rflow'` does not have such requirement. In other words, it is easier to prepare training data for `'maisi3d-rflow'`.\n", "- For the released model weights, `'maisi3d-rflow'` can generate images with better quality for head region and small output volumes, and comparable quality for other cases compared with `'maisi3d-ddpm'`." ] }, { "cell_type": "code", "execution_count": 3, "id": "3d233abe-d69c-4b57-9655-33c2c3da6c96", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[2025-03-14 16:29:13.938][ INFO](notebook) - MAISI version is maisi3d-rflow, whether to use body_region is False\n" ] } ], "source": [ "maisi_version = \"maisi3d-rflow\"\n", "if maisi_version == \"maisi3d-ddpm\":\n", " model_def_path = \"./configs/config_maisi3d-ddpm.json\"\n", "elif maisi_version == \"maisi3d-rflow\":\n", " model_def_path = \"./configs/config_maisi3d-rflow.json\"\n", "else:\n", " raise ValueError(f\"maisi_version has to be chosen from ['maisi3d-ddpm', 'maisi3d-rflow'], yet got {maisi_version}.\")\n", "with open(model_def_path, \"r\") as f:\n", " model_def = json.load(f)\n", "include_body_region = model_def[\"include_body_region\"]\n", "logger.info(f\"MAISI version is {maisi_version}, whether to use body_region is {include_body_region}\")" ] }, { "cell_type": "markdown", "id": "671e7f10", "metadata": {}, "source": [ "## Step 1: Training Data Preparation\n" ] }, { "cell_type": "markdown", "id": "d8e29c23", "metadata": {}, "source": [ "### Simulate a special dataset\n", "\n", "It is widely recognized that training AI models is a time-intensive process. In this instance, we will simulate a small dataset and conduct training over multiple epochs. While the performance may not reach optimal levels due to the abbreviated training duration, the entire pipeline will be completed within minutes.\n", "\n", "`sim_datalist` provides the information of the simulated datasets. It lists 2 training images. The size of the dimension is defined by the `sim_dim`.\n", "\n", "The diffusion model and ControlNet utilize a U-shaped convolutional neural network architecture, requiring matching input and output dimensions. Therefore, it is advisable to resample the input image dimensions to be multiples of 2 for compatibility. In this case, we have chosen dimensions that are multiples of 128.\n", "\n", "The training workflow requires one JSON file to specify the image embedding and segmentation pairs. In addtional, the diffusion model used in ControlNet necessitates additional input attributes, including output dimension, output spacing, and top/bottom body region. The dimensions, and spacing can be extracted from the header information of the training images. The pseudo whole-body segmentation mask, and the top/bottom body region inputs can be determined through manual examination or by utilizing segmentation masks from tools such as [TotalSegmentator](https://github.com/wasserth/TotalSegmentator) or [MONAI VISTA](https://github.com/Project-MONAI/VISTA). The body regions are formatted as 4-dimensional one-hot vectors: the head and neck region is represented by [1,0,0,0], the chest region by [0,1,0,0], the abdomen region by [0,0,1,0], and the lower body region (below the abdomen) by [0,0,0,1]. \n", "\n", "To train the ControlNet/diffusion unet, we first store the latent features (image embeddings) produced by the autoencoder's encoder in local storage. This allows the latent diffusion model to directly utilize these features, thereby conserving both time and GPU memory during the training process. Additionally, we have provided the script for multi-GPU processing to save latent features from all training images, significantly accelerating the creation of the entire training set. Please check the Step 1 Create Training Data in [maisi_diff_unet_training_tutorial](./maisi_diff_unet_training_tutorial.ipynb) and [diff_model_create_training_data.py](./scripts/diff_model_create_training_data.py) for how to encode images and save as image embeddings.\n", "\n", "The JSON file used in ControlNet training has the following structure:" ] }, { "cell_type": "code", "execution_count": null, "id": "fc32a7fe", "metadata": {}, "outputs": [], "source": [ "sim_dim = [256, 256, 128]\n", "sim_datalist = {\n", " \"training\": [\n", " {\n", " \"image\": \"tr_image_001_emb.nii.gz\", # relative path to the image embedding file\n", " # relative path to the combined label (pseudo whole-body segmentation mask + ROI mask) file\n", " \"label\": \"tr_label_001.nii.gz\",\n", " \"fold\": 0, # fold index for cross validation. If the dataset item's fold value is equal to the fold value\n", " # in config_maisi_controlnet_train.json, then it is used for validation. Otherwise, it is used for training.\n", " # In the current parameters, fold 0 is for validation.\n", " \"dim\": sim_dim, # the dimension of image\n", " \"spacing\": [1.5, 1.5, 1.5], # the spacing of image\n", " },\n", " {\n", " \"image\": \"tr_image_002_emb.nii.gz\",\n", " \"label\": \"tr_label_002.nii.gz\",\n", " \"fold\": 1,\n", " \"dim\": sim_dim,\n", " \"spacing\": [1.5, 1.5, 1.5],\n", " },\n", " {\n", " \"image\": \"tr_image_003_emb.nii.gz\",\n", " \"label\": \"tr_label_003.nii.gz\",\n", " \"fold\": 1,\n", " \"dim\": sim_dim,\n", " \"spacing\": [1.5, 1.5, 1.5],\n", " },\n", " ]\n", "}\n", "if include_body_region:\n", " for i in range(len(sim_datalist[\"training\"])):\n", " # body region index\n", " sim_datalist[\"training\"][i][\"top_region_index\"] = [0, 1, 0, 0] # the top region index of the image\n", " sim_datalist[\"training\"][i][\"bottom_region_index\"] = [0, 0, 0, 1] # the bottom region index of the image" ] }, { "cell_type": "markdown", "id": "b9ac7677", "metadata": {}, "source": [ "### Generate simulated images and labels\n", "\n", "Now we can use MONAI `create_test_image_3d` and `nib.Nifti1Image` functions to generate the 3D simulated images under the `work_dir`." ] }, { "cell_type": "code", "execution_count": 5, "id": "1b199078", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[2025-03-14 16:29:13.952][ INFO](notebook) - Save data list json file to ./temp_work_dir_controlnet_train_demo/sim_datalist.json\n", "[2025-03-14 16:29:16.033][ INFO](notebook) - Generated simulated images.\n", "[2025-03-14 16:29:16.034][ INFO](notebook) - img_emb shape: (64, 64, 32, 4)\n", "[2025-03-14 16:29:16.035][ INFO](notebook) - label shape: (256, 256, 128)\n" ] } ], "source": [ "work_dir = \"./temp_work_dir_controlnet_train_demo\"\n", "if not os.path.isdir(work_dir):\n", " os.makedirs(work_dir)\n", "\n", "dataroot_dir = os.path.join(work_dir, \"sim_dataroot\")\n", "if not os.path.isdir(dataroot_dir):\n", " os.makedirs(dataroot_dir)\n", "\n", "datalist_file = os.path.join(work_dir, \"sim_datalist.json\")\n", "with open(datalist_file, \"w\") as f:\n", " json.dump(sim_datalist, f, indent=4)\n", "logger.info(f\"Save data list json file to {datalist_file}\")\n", "\n", "for d in sim_datalist[\"training\"]:\n", " # The image embedding is downsampled twice by Autoencoder.\n", " img_emb, _ = create_test_image_3d(\n", " sim_dim[0] // 4,\n", " sim_dim[1] // 4,\n", " sim_dim[2] // 4,\n", " rad_max=10,\n", " num_seg_classes=1,\n", " random_state=np.random.RandomState(42),\n", " )\n", " # The label has a same shape as the original image.\n", " _, label = create_test_image_3d(\n", " sim_dim[0], sim_dim[1], sim_dim[2], rad_max=10, num_seg_classes=1, random_state=np.random.RandomState(42)\n", " )\n", "\n", " image_fpath = os.path.join(dataroot_dir, d[\"image\"])\n", " # We repeat the volume 4 times to simulate the channel dimension of latent features.\n", " img_emb = np.stack([img_emb] * 4, axis=3)\n", " nib.save(nib.Nifti1Image(img_emb, affine=np.eye(4)), image_fpath)\n", " label_fpath = os.path.join(dataroot_dir, d[\"label\"])\n", " nib.save(nib.Nifti1Image(label, affine=np.eye(4)), label_fpath)\n", "\n", "logger.info(\"Generated simulated images.\")\n", "logger.info(f\"img_emb shape: {img_emb.shape}\")\n", "logger.info(f\"label shape: {label.shape}\")" ] }, { "cell_type": "markdown", "id": "19724631", "metadata": {}, "source": [ "## Step 2: Training Config Preparation" ] }, { "cell_type": "markdown", "id": "c2389853", "metadata": {}, "source": [ "### Set up directories and configurations\n", "\n", "To optimize the demonstration for time efficiency, we have adjusted the training epochs to 2. Additionally, we modified the `num_splits` parameter in [AutoencoderKlMaisi](https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/generation/maisi/networks/autoencoderkl_maisi.py#L881) from its default value of 16 to 4. This adjustment reduces the spatial splitting of feature maps in convolutions, which is particularly beneficial given the smaller input size. (This change helps convert convolutions to a for-loop based approach, thereby conserving GPU memory resources.)" ] }, { "cell_type": "code", "execution_count": 6, "id": "6c7b434c", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[2025-03-14 16:29:16.049][ INFO](notebook) - files and folders under work_dir: ['config_maisi.json', 'models', 'config_maisi_controlnet_train.json', 'outputs', 'sim_dataroot', '.ipynb_checkpoints', 'environment_maisi_controlnet_train.json', 'sim_datalist.json'].\n", "[2025-03-14 16:29:16.050][ INFO](notebook) - number of GPUs: 1.\n" ] } ], "source": [ "env_config_path = \"./configs/environment_maisi_controlnet_train.json\"\n", "train_config_path = \"./configs/config_maisi_controlnet_train.json\"\n", "\n", "# Load environment configuration, model configuration and model definition\n", "with open(env_config_path, \"r\") as f:\n", " env_config = json.load(f)\n", "\n", "with open(train_config_path, \"r\") as f:\n", " train_config = json.load(f)\n", "\n", "with open(model_def_path, \"r\") as f:\n", " model_def = json.load(f)\n", "\n", "env_config_out = copy.deepcopy(env_config)\n", "train_config_out = copy.deepcopy(train_config)\n", "model_def_out = copy.deepcopy(model_def)\n", "\n", "# Set up directories based on configurations\n", "env_config_out[\"data_base_dir\"] = dataroot_dir\n", "env_config_out[\"json_data_list\"] = datalist_file\n", "env_config_out[\"model_dir\"] = os.path.join(work_dir, env_config_out[\"model_dir\"])\n", "env_config_out[\"output_dir\"] = os.path.join(work_dir, env_config_out[\"output_dir\"])\n", "env_config_out[\"tfevent_path\"] = os.path.join(work_dir, env_config_out[\"tfevent_path\"])\n", "# We don't load pretrained checkpoints for demo\n", "env_config_out[\"trained_autoencoder_path\"] = None\n", "env_config_out[\"trained_diffusion_path\"] = None\n", "env_config_out[\"trained_controlnet_path\"] = None\n", "env_config_out[\"exp_name\"] = \"tutorial_training_example\"\n", "\n", "\n", "# Create necessary directories\n", "os.makedirs(env_config_out[\"model_dir\"], exist_ok=True)\n", "os.makedirs(env_config_out[\"output_dir\"], exist_ok=True)\n", "os.makedirs(env_config_out[\"tfevent_path\"], exist_ok=True)\n", "\n", "env_config_filepath = os.path.join(work_dir, \"environment_maisi_controlnet_train.json\")\n", "with open(env_config_filepath, \"w\") as f:\n", " json.dump(env_config_out, f, sort_keys=True, indent=4)\n", "\n", "# Update training configuration for demo\n", "max_epochs = 2\n", "train_config_out[\"controlnet_train\"][\"n_epochs\"] = max_epochs\n", "# We disable weighted_loss for dummy data, which is used to apply more penalty\n", "# to the region of interest (e.g., tumors). When weighted_loss=1,\n", "# we treat all regions equally in loss computation.\n", "train_config_out[\"controlnet_train\"][\"weighted_loss\"] = 1\n", "# We also set weighted_loss_label to None, which indicates the list of label indices that\n", "# we want to apply more penalty during training.\n", "train_config_out[\"controlnet_train\"][\"weighted_loss_label\"] = [None]\n", "# We set it as a small number for demo\n", "train_config_out[\"controlnet_infer\"][\"num_inference_steps\"] = 1\n", "\n", "train_config_filepath = os.path.join(work_dir, \"config_maisi_controlnet_train.json\")\n", "with open(train_config_filepath, \"w\") as f:\n", " json.dump(train_config_out, f, sort_keys=True, indent=4)\n", "\n", "# Update model definition for demo\n", "model_def_out[\"autoencoder_def\"][\"num_splits\"] = 4\n", "model_def_filepath = os.path.join(work_dir, \"config_maisi.json\")\n", "with open(model_def_filepath, \"w\") as f:\n", " json.dump(model_def_out, f, sort_keys=True, indent=4)\n", "\n", "# Print files and folders under work_dir\n", "logger.info(f\"files and folders under work_dir: {os.listdir(work_dir)}.\")\n", "\n", "# Adjust based on the number of GPUs you want to use\n", "num_gpus = 1\n", "logger.info(f\"number of GPUs: {num_gpus}.\")" ] }, { "cell_type": "code", "execution_count": 7, "id": "95ea6972", "metadata": {}, "outputs": [], "source": [ "def run_torchrun(module, module_args, num_gpus=1):\n", " # Define the arguments for torchrun\n", " num_nodes = 1\n", "\n", " # Build the torchrun command\n", " torchrun_command = [\n", " \"torchrun\",\n", " \"--nproc_per_node\",\n", " str(num_gpus),\n", " \"--nnodes\",\n", " str(num_nodes),\n", " \"-m\",\n", " module,\n", " ] + module_args\n", "\n", " # Set the OMP_NUM_THREADS environment variable\n", " env = os.environ.copy()\n", " env[\"OMP_NUM_THREADS\"] = \"1\"\n", "\n", " # Execute the command\n", " process = subprocess.Popen(torchrun_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, env=env)\n", "\n", " # Print the output in real-time\n", " try:\n", " while True:\n", " output = process.stdout.readline()\n", " if output == \"\" and process.poll() is not None:\n", " break\n", " if output:\n", " print(output.strip())\n", " except Exception as e:\n", " print(f\"An error occurred: {e}\")\n", " finally:\n", " # Capture and print any remaining output\n", " stdout, stderr = process.communicate()\n", " print(stdout)\n", " if stderr:\n", " print(stderr)\n", " return" ] }, { "cell_type": "markdown", "id": "e81a9e48", "metadata": {}, "source": [ "## Step 3: Train the Model\n", "\n", "After all latent feature/mask pairs have been created, we will initiate the multi-GPU script to train ControlNet.\n", "\n", "The image generation process utilizes the [DDPM scheduler](https://arxiv.org/pdf/2006.11239) with 1,000 iterative steps. The ControlNet is optimized using L1 loss and a decayed learning rate scheduler. The batch size for this process is set to 1." ] }, { "cell_type": "code", "execution_count": 8, "id": "ade6389d", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[2025-03-14 16:29:16.061][ INFO](notebook) - Training the model...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-03-14 16:29:23.336][ INFO](maisi.controlnet.training) - Number of GPUs: 8\n", "[2025-03-14 16:29:23.336][ INFO](maisi.controlnet.training) - World_size: 1\n", "[2025-03-14 16:29:24.771][ INFO](maisi.controlnet.training) - trained diffusion model is not loaded.\n", "[2025-03-14 16:29:24.771][ INFO](maisi.controlnet.training) - set scale_factor -> 1.0.\n", "2025-03-14 16:29:25,271 - INFO - 'dst' model updated: 180 of 231 variables.\n", "[2025-03-14 16:29:25.277][ INFO](maisi.controlnet.training) - train controlnet model from scratch.\n", "[2025-03-14 16:29:25.300][ INFO](maisi.controlnet.training) - total number of training steps: 4.0.\n", "[2025-03-14 16:29:26.826][ INFO](maisi.controlnet.training) -\n", "[Epoch 1/2] [Batch 1/2] [LR: 0.00000563] [loss: 0.8278] ETA: 0:00:01.523338\n", "[2025-03-14 16:29:26.974][ INFO](maisi.controlnet.training) -\n", "[Epoch 1/2] [Batch 2/2] [LR: 0.00000250] [loss: 0.8289] ETA: 0:00:00\n", "[2025-03-14 16:29:27.585][ INFO](maisi.controlnet.training) - best loss -> 0.8283329606056213.\n", "[2025-03-14 16:29:28.909][ INFO](maisi.controlnet.training) -\n", "[Epoch 2/2] [Batch 1/2] [LR: 0.00000063] [loss: 0.8288] ETA: 0:00:01.934548\n", "[2025-03-14 16:29:29.052][ INFO](maisi.controlnet.training) -\n", "[Epoch 2/2] [Batch 2/2] [LR: 0.00000000] [loss: 0.8277] ETA: 0:00:00\n", "[2025-03-14 16:29:29.716][ INFO](maisi.controlnet.training) - best loss -> 0.8282470703125.\n", "\n" ] } ], "source": [ "logger.info(\"Training the model...\")\n", "\n", "# Define the arguments for torchrun\n", "module = \"scripts.train_controlnet\"\n", "module_args = [\n", " \"--environment-file\",\n", " env_config_filepath,\n", " \"--config-file\",\n", " model_def_filepath,\n", " \"--training-config\",\n", " train_config_filepath,\n", "]\n", "\n", "run_torchrun(module, module_args, num_gpus=num_gpus)" ] }, { "cell_type": "markdown", "id": "cc3c996d", "metadata": {}, "source": [ "## Step 4: Model Inference\n", "\n", "Upon completing the training of the ControlNet, we can employ the multi-GPU script to perform inference. \n", "By integrating autoencoder, diffusion model, and controlnet, this process will generate 3D images with specified top/bottom body regions, spacing, and dimensions based on input masks." ] }, { "cell_type": "code", "execution_count": 9, "id": "936360c8", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[2025-03-14 16:29:32.229][ INFO](notebook) - Inference...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-03-14 16:29:39.519][ INFO](maisi.controlnet.infer) - Number of GPUs: 8\n", "[2025-03-14 16:29:39.519][ INFO](maisi.controlnet.infer) - World_size: 1\n", "[2025-03-14 16:29:39.990][ INFO](maisi.controlnet.infer) - trained autoencoder model is not loaded.\n", "[2025-03-14 16:29:41.213][ INFO](maisi.controlnet.infer) - trained diffusion model is not loaded.\n", "[2025-03-14 16:29:41.213][ INFO](maisi.controlnet.infer) - set scale_factor -> 1.0.\n", "2025-03-14 16:29:41,716 - INFO - 'dst' model updated: 180 of 231 variables.\n", "[2025-03-14 16:29:41.721][ INFO](maisi.controlnet.infer) - trained controlnet is not loaded.\n", "[2025-03-14 16:29:42.102][ INFO](root) - `controllable_anatomy_size` is not provided.\n", "[2025-03-14 16:29:42.104][ INFO](root) - ---- Start generating latent features... ----\n", "[2025-03-14 16:29:42.670][ INFO](root) - ---- DM/ControlNet Latent features generation time: 0.565190315246582 seconds ----\n", "[2025-03-14 16:29:42.672][ INFO](root) - ---- Start decoding latent features into images... ----\n", "[2025-03-14 16:29:43.314][ INFO](root) - ---- Image VAE decoding time: 0.6416211128234863 seconds ----\n", "2025-03-14 16:29:43,602 INFO image_writer.py:197 - writing: temp_work_dir_controlnet_train_demo/outputs/sample_20250314_162943_586788_image.nii.gz\n", "2025-03-14 16:29:43,940 INFO image_writer.py:197 - writing: temp_work_dir_controlnet_train_demo/outputs/sample_20250314_162943_586788_label.nii.gz\n", "\n", "\n", " 0%| | 0/1 [00:00" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(Image(filename=\"./figures/maisi_control-train-examples.png\"))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 5 }