{"id":18485,"date":"2024-11-29T09:38:39","date_gmt":"2024-11-29T07:38:39","guid":{"rendered":"https:\/\/www.btc-embedded.com\/?p=18485"},"modified":"2024-11-29T10:02:25","modified_gmt":"2024-11-29T08:02:25","slug":"how-to-automate-dependency-management-part-1","status":"publish","type":"post","link":"https:\/\/www.btc-embedded.com\/zh-hans\/how-to-automate-dependency-management-part-1\/","title":{"rendered":"How to Automate Dependency Management: Part 1"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Very few people develop their software from scratch. Why would they, after all, when so many others have already done the work for almost any algorithm you could think of? Rather than re-invent the wheel, it\u2019s more efficient (and more fun) to work on the novel tasks in front of you and let someone else handle the technical details with a quick \u2018import\u2019.<\/p>

But this strength of re-using software comes with a very great weakness: dependency management. What do you do when the module you\u2019re importing upgrades to a new version? How do you ensure that your own code still works, and there are no unintended side effects of the upgrade?<\/p>

Most people just handle these upgrades manually, and because of the effort that can be involved, this often means their dependencies are often several versions behind the latest release. This isn\u2019t necessarily a bad thing, but it does mean you\u2019re missing out on bugfixes, performance improvements, and new features. And you don\u2019t need to! With some simple workflow automation, you can reap the benefits of CI\/CD best practices and let your process work for you!<\/p>

In this article<\/strong>, I\u2019ll explain current best practices for dependency management in an MBD context. By the end of it, you\u2019ll have enough familiarity with the process as whole to be able to implement your own production CI pipeline for managing, migrating, and testing dependencies! You\u2019ll also have set up your own sample dependency migration pipeline, so you have a practical example to reference.<\/p>

In part 1 of this article<\/strong> (you are here!), I\u2019ll focus on the development part of the toolchain, where we restructure an original monolithic model to version its dependencies and build them on-the-fly. In part 2<\/a> of this article<\/strong>, I\u2019ll add in a CI pipeline to automatically migrate and test our dependencies to the latest version. Here\u2019s an outline of the steps we\u2019ll be going through:<\/p>

  1. Creating our original, unified model<\/strong><\/li>
  2. Separating out our model into main module and dependency module<\/strong><\/li>
  3. Serving our dependency module through a package-manager<\/strong><\/li>
  4. Migrating our main module to use the latest version of its dependency<\/li>
  5. Testing the main module to validate the migration didn\u2019t break anything<\/li><\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    Step 1: Creating our Model <\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    First things first, we need some models! I\u2019m going to be shamelessly stealing the example my coworker, Thabo, has already developed, and build up from there. If you\u2019re curious, the git repos have the fully-fledged example all worked out, but I\u2019m only going to be grabbing the \u2018core\u2019 files to begin with. If you want to follow along, navigate<\/a> to the first repo, and grab the swc_1.slx and start.m files (these files, by the way, were generated for MATLAB R2023b) from inside the src folder.<\/p>

    These two files will define my main unit, but they won\u2019t run unless I also get the dependency library along with them. Navigate<\/a> to the other git repo, and download the shared_module_a.slx inside the src folder, placing it in the same directory as the model from above.<\/p>

    Just to make sure nothing went wrong, go ahead and run start.m in your Matlab workspace, and then open and simulate swc_1:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t
    Happy as a clam!<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    Step 2: Separating Out our Model <\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Now that we know they work together, let\u2019s separate them, so that they can be versioned independently. You can delete any cache folders created by Simulink, and then structure the models into two separate folders<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    And from here, I\u2019m going to create two separate git repos and commit them both. First, create the repositories in GitHub, using the web interface (they can be uninitialized\/empty, and mine are set to \u2018private\u2019). Then, for both the dep folder and swc1 folders, initialize and commit them to their respective repositories:<\/p>

    git init<\/span><\/p>

    git add .<\/span><\/p>

    git commit \u2013m \u201cinitial model commit\u201d<\/span><\/p>

    git remote add origin https:\/\/github.com\/btc-nathand\/swc1.git<\/span><\/p>

    git branch -M main<\/span><\/p>

    git push -u origin main<\/span><\/p>

    Replacing the URL in both cases with the appropriate link to the repository you just created in the GitHub web interface.<\/p>

    Success! We\u2019ve broken the link between our main SWC and its dependent!<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t

    Step 3: Serving the Dependency to the Main Module <\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
    \n\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Now that we\u2019ve separated the two models, we\u2019re going to want to also have some way to bring them back together again\u2014otherwise we\u2019ve just broken everything, with no benefit to anyone! We\u2019ll do this by treating the dependent as an independently developed software module\u2014and that means we need to set up an artifact repository server to make the file available to other modules, so our main unit can ask for the dependencies it needs.<\/p>

    At a high level, this is the intended workflow we\u2019re looking to set up in this step: <\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Developer 1 and Developer 2 work on their codebases separately. When Developer 1 is finished, he releases his code publicly; this lets Developer 2 pull it in as a dependency to her own code. This lets Developer 2 have strict separation of concerns between code she\u2019s responsible for and code she\u2019s just using, and also gives a way for her to explicitly track which version of dependent code she needs.<\/p>

    Having a way to separate out the development environment (my code!) from the dependency source (your code!) is vital here. While we could use something like git submodules, this would blur the line between responsibilities, and we still could (accidently or maliciously!) push unintended changes to the dependency. What we need isn\u2019t just a way to import code from one place to another\u2014we need a full-blown way of versioning and serving code as a \u2018black-box\u2019 with strict controls between unrelated projects. What we need is a package manager!<\/p>

    There are lots of mature, maintained package managers out there, and any one of those will work. One of the more well-known ones is JFrog\u2019s Artifactory, which is a dual-bundle artifact serving tool with an environment definition tool called Conan, and I\u2019ll be using these in this tutorial. First, download the latest<\/a> Artifactory installer (on a real production workflow, this would be hosted on a separate server, but running it locally is perfectly fine for this example). Navigate into the app\/bin directory and double-click on \u201cartifactory.bat\u201d to start up the service. Go into your browser and navigate to http:\/\/localhost:8081<\/a>, where you should see a pretty, green welcome screen. Log in with the default username (admin) and password (password), and you\u2019ll be prompted through some basic configuration options. Leave everything as default, and finally you\u2019ll be greeted with the main loading page:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    From here, we\u2019re going to create a new repository to host and serve our dependency module. Configure it as a local Conan project and give it a unique key. Because we\u2019re running all this locally, we don\u2019t need to set up any user management permissions, but again, this would be helpful in a production environment.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Switch back to your IDE. To connect our git, where our dependency is stored, with Artifactory, where it\u2019s served, we have to create a conanfile in the dependency module file structure. Create a file named conanfile.py, and put the following text into it, replacing the URL with your own repo. If you\u2019re curious about the configuration options available here, you can read more on the Conan website<\/a>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    from conan import ConanFile<\/span><\/p>

    from conan.tools.files import copy<\/span><\/p>

    class MyPackage(ConanFile):<\/span><\/p>

    \u00a0\u00a0\u00a0 name = “shared_module_a”<\/span><\/p>

    \u00a0\u00a0\u00a0 version = “1.0.0”<\/span><\/p>

    \u00a0\u00a0\u00a0 license = “MIT”<\/span><\/p>

    \u00a0\u00a0\u00a0 url = “https:\/\/github.com\/btc-nathand\/dep”<\/span><\/p>

    \u00a0\u00a0\u00a0 description = “Conan package for Shared Module A”<\/span><\/p>

    \u00a0\u00a0\u00a0 exports_sources = “*.slx”<\/span><\/p>

    \u00a0\u00a0\u00a0 no_copy_source = False<\/span><\/p>

    \u00a0\u00a0\u00a0 def package(self):<\/span><\/p>

    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 copy(self, “*.slx”, self.source_folder, self.package_folder, keep_path=False)<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Save the file and open a terminal to finish the packaging. First, let\u2019s do some restructuring, so that our project isn\u2019t just a flat folder:<\/p>

    cd dep<\/span><\/p>

    mkdir src<\/span><\/p>

    mv shared_module_a src<\/span><\/p>

    Of course, we need to install Conan to have it available. We\u2019re also going to need to finish our configuration by telling Conan where our dependency is supposed to go when we release it, and then finalize our changes and push them as our first commit:<\/p>

    pip install conan<\/span><\/p>

    conan profile detect<\/span><\/p>

    conan remote add artifactory http:\/\/localhost:8081\/artifactory\/api\/conan\/dep-repo-conan<\/span><\/p>

    conan create .<\/span><\/p>

    conan upload shared_module_a -r=artifactory<\/span><\/p>

    And with that we\u2019ve successfully uploaded our dependency! This means it\u2019s available for anyone to download from our server; the next step is to tell our main SWC how to find and install it, so that we can create a complete running module.<\/p>

    We\u2019ll need to go back into our main SWC, and create a conanfile, but instead of uploading to the Artifactory repo, we\u2019ll point it to pull and merge in the dependency with our own model at build time. Create another conanfile.py, with the following contents:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    from os.path import join<\/span><\/p>

    from conan import ConanFile<\/span><\/p>

    from conan.tools.env import VirtualBuildEnv, VirtualRunEnv<\/span><\/p>

    from conan.tools.files import copy<\/span><\/p>

    class SWC1(ConanFile):<\/span><\/p>

    \u00a0\u00a0\u00a0 name = “swc_1”<\/span><\/p>

    \u00a0\u00a0\u00a0 version = “1.0.0”<\/span><\/p>

    \u00a0\u00a0\u00a0 license = “MIT”<\/span><\/p>

    \u00a0\u00a0\u00a0 url = “https:\/\/github.com\/btc-nathand\/swc1”<\/span><\/p>

    \u00a0\u00a0\u00a0 description = “Conan package for SWC1”<\/span><\/p>

    \u00a0\u00a0\u00a0 requires = “shared_module_a\/1.0.0”<\/span><\/p>

    \u00a0\u00a0\u00a0 def generate(self):<\/span><\/p>

    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # disable generation of scripts in the build folder<\/span><\/p>

    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 VirtualBuildEnv(self)<\/span><\/p>

    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 VirtualRunEnv(self)<\/span>\u00a0<\/span><\/p>

    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # copy model files from dependencies root folder into<\/span><\/p>

    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # folders with the dep-name inside the build_folder<\/span><\/p>

    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 for dep in self.dependencies.values():<\/span><\/p>

    \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 copy(self, “*.slx”, dep.package_folder, join(self.build_folder, ‘shared’))<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    We\u2019ll want this at the root level of swc1 to define our project. Then, much like with the dependency, we\u2019ll restructure our main module project hierarchy, and create the conan project:<\/p>

    cd swc1<\/span><\/p>

    mkdir src<\/span><\/p>

    mv swc_1.slx src<\/span><\/p>

    mv start.m src<\/span><\/p>

    conan create .<\/span><\/p>

    Finally, tell Conan to pull in the dependencies it needs from our Artifactory server:<\/p>

    conan install . -r artifactory<\/span><\/p>

    Just to make sure that everything is pieced together, I\u2019m going to look at my workspace and confirm that my missing dependency did, in fact, get downloaded:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    You\u2019ll notice that now, in addition to the files we added in and moved around by hand, we have a \u2018shared\u2019 folder next the \u2018src\u2019, which contains our resolved dependency. Great! Our Artifactory server is doing its job. To make sure that this doesn\u2019t get checked into git (it\u2019s supposed to be resolved at build time\u2014ideally, a local initialization script will run our \u2018conan install\u2019 command along with doing a git pull to make sure everything is up to date), lets pop in a quick .gitignore:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    # ignore anything from the simulink cache & all generated files<\/span><\/p>

    *.slxc<\/span><\/p>

    *_ert_rtw\/<\/span><\/p>

    slprj\/<\/span><\/p>

    *.html<\/span><\/p>

    # ignore shared modules that are added by the package manager<\/span><\/p>

    shared\/<\/span><\/p>

    conan.log<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    You can go ahead and copy over the same .gitignore into the dep repo as well.<\/p>

    Great! We’ve just finished setting up our system so that it can successfully upload and resolve dependencies. Now\u2019s a good point to save everything through a git upload.<\/p>

    We\u2019ve done a lot, so let\u2019s summarize our status so far. From a developer perspective, everything\u2019s been finished:<\/p>

    – The maintainer of dep, when he\u2019s done making his changes, in addition to checking in to git (for internal maintenance), also makes them available to other teams through Artifactory (for external linkage), by doing a \u2018conan upload\u2019<\/p>

    – The maintainer of SWC1, when she\u2019s ready to develop her software, simply declares her dependencies, and then runs a \u2018conan install\u2019 to bring all the defined dependency versions into her \u2018shared\u2019 folder<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

    Very few people develop their software from scratch. Wh […]<\/p>\n","protected":false},"author":16,"featured_media":19068,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"elementor_theme","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[124,53,123,48,122,51,80,121],"product":[],"use_cases":[],"class_list":["post-18485","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-embedded-coder","tag-iso-26262","tag-matlab","tag-simulink","tag-standard-tools","tag-targetlink","tag-test-generation","tag-testing"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/posts\/18485","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/comments?post=18485"}],"version-history":[{"count":39,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/posts\/18485\/revisions"}],"predecessor-version":[{"id":19079,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/posts\/18485\/revisions\/19079"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/media\/19068"}],"wp:attachment":[{"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/media?parent=18485"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/categories?post=18485"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/tags?post=18485"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/product?post=18485"},{"taxonomy":"use_cases","embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/use_cases?post=18485"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}