This report will be a little bit longer than normal cause it will comprehend the last week changes too. So let’s start.

  1. Status:: Currently engaged with the path where each new_version.json file will be saved (see #c) and also to rebuild update_nodes_with_new_versions to make it compatible with each new file (that will be loaded on these new paths).

  2. Abstract:: Last meeting, 02/06 I’ve discussed to change the format how I was saving the output versions to make it compatible with the original graph changes used by update_upstream_versions. And also asked about some doubts regarding the make_graph.py script, how it works and some key words used during the code: like our common feedstocks as also some new ones (at least for me) called outputs and stubs. There were others important questions regarding the construction and recontruin of nodes inside the running of make_graph. All this questions triggereed by this statement inside it:

        # handle multi outputs
    if "outputs" in yaml_dict:
   sub_graph["outputs_names"] = sorted(
       list({d.get("name", "") for d in yaml_dict["outputs"]}),
   )
    gx.graph["outputs_lut"] = outputs_lut
    strong_exports = {
   node_name
   for node_name, node in gx.nodes.items()
   if node.get("payload").get("strong_exports", False)
    }

    Not exactly by the meaning of these structures, but how it interferes of the nodes architecture, cause in my initial guess I thought that there are a separation inside the graph of the feedstocks nodes and the other ones (well actually this separation exists, but only our feedstocks packages are actually stored for use as a node). Then, as there aren’t any kind of separation, there will be no problema when I executed a loop like below (gladly, everything is still just a node).

     to_update = []
 _all_nodes = [t for t in gx.nodes.items()]
    for node, node_attrs in _all_nodes:
   attrs = node_attrs["payload"]
   if attrs.get("bad") or attrs.get("archived"):
       attrs["new_version"] = False
       continue
   to_update.append((node, attrs))

    So I can work peacefully with the current data output format I was imagining something like this: {"node_name":{"bad":"", "new_version":""}}

I also added the keys versions_erros and version_attempts that I thought been related with the get_version function, but I was wrong it’s related with the mitigation phase of ‘the not’. So I didn’t need to use this information, at least here. As we can see above, I also separated another loop for the packages marked as bad or archived that will not be loaded during the get new version phase. Them now, we have:

 up_to = {}
 for node, node_attrs in to_update:
        # checking each node
        with node_attrs as attrs:
            up_to = {}

            # verify the actual situation of the package;
            actual_ver = str(attrs.get("version"))

            # rude exception
            if node == "ca-policy-lcg":
                up_to["ca-policy-lcg"] = {
                    "bad": attrs.get("bad"),
                    "new_version": False
                }
                Node_count += 1
                continue

            # New verison request
            try:
                new_version = get_latest_version(node, attrs, sources)
                attrs["new_version"] = new_version or attrs["new_version"]
            except Exception as e:
                try:
                    se = repr(e)
                except Exception as ee:
                    se = "Bad exception string: {}".format(ee)
                logger.warning(
                    f"Warning: Error getting upstream version of {node}: {se}"
                )
                attrs["bad"] = "Upstream: Error getting upstream version"
            else:
                logger.info(
                    f"# {Node_count:<5} - {node} - {attrs.get('version')} - {attrs.get('new_version')}",
                )
            up_to[f"{node}"] = {
                "bad": attrs.get("bad"),
                "new_version": attrs.get("new_version")
            }
            Node_count += 1

Returning as output the up_to dictionary of each update node. Which will be save into a json file called new_version.json at main:

def main(args: Any = None) -> None:
    if CONDA_FORGE_TICK_DEBUG:
        setup_logger(logger, level="debug")
    else:
        setup_logger(logger)

    logger.info("Reading graph")
    # Graph enabled for inspection
    gx = load_graph()

    # call update
    to_update = new_update_upstream_versions(gx)

    logger.info("writing out file")
    with open("new_version.json", "w") as outfile:
        json.dump(to_update, outfile)

Them after this alterations a come with the make_graph, cause now (as commented by CJ #CJ) we need these new information been write at our graph (maintaining a separate process) so we choose make_graph to facilitate our work (it’s easier to work with a rebuild of the graph then work with all the dependencies that are related with update_upstream_versions). This attempt to insert the information created a new function inside it called update_nodes_with_new_versions that updates the graph using our new_version.json file

def update_nodes_with_new_versions(gx):
    """Updates every node with it's new version (when available)"""
    try:
        with open('new_version.json') as file:
            ver_attrs = json.load(file)
    except FileNotFoundError as e:
        logger.debug(f'Process interrupted with exeption: {e}')
        return
    
    # Update graph according to ver_attrs
    for node, vertrs in ver_attrs.items():
        with gx.nodes[f'{node}']['payload'] as attrs:
            if vertrs["bad"]:
                attrs["bad"] = vertrs.get("bad")
            elif vertrs["archived"]:
                attrs["archived"] = vertrs.get("archived")
            # Update with new information
            attrs["new_version"] = vertrs.get("new_version")
  1. Next Steps:: As talked before, I’ve removed the non necessary information regarding the migrations keys (“new_version_attempts” and “new_version_errors”). So for now, we should dump one file per feedstock of the “new versions.json” as explained by Beckermr, then work with circleci to test this new approach, we’ve talked about a new job or rebuild one to make everything operational (probably). Some discussion regarding the make_graph will be available at a separate post here this week with more other informations. As also we can see at p1027 that I am expecting to rename the actual new_update_version function to update_upstream_version and remove the old one (for compatibility) and the do the new updates with make_graphas a new PR.

Observation, there is a very exciting talk with Christopher Wright (one of my Conda-forge mentors) about conda-forge and the auto-tick-bot, named “The Automation of Conda-Forge” that you can watch at AnacondaCon !