MDEV-35565 Server crashes simplifying group by <subquery>

[mariadb-RexJohnston]

During optimization, we may call update_depend_map_for_order() on a group by list containing an expression with an outer reference. The table map for this outer reference will be collected by Item_subselect::recalc_used_tables and used to return a depend_map. This map will indicate the presence of tables not in this select and can cause erroneous results, poor execution plans or crashes. We update the class used to collect these bitmaps to check whether the containing select_lex is applicable, and if not, set the OUTER_REF_TABLE_BIT bit in the table map.

[gemini-code-assist]

Code Review

This pull request addresses a server crash (MDEV-35565) when simplifying queries with a GROUP BY clause containing a subquery. The fix updates Field_fixer::visit_field in sql/item_subselect.cc to correctly identify outer references by traversing merged select structures. The review feedback points out a potential null pointer dereference in the new logic if item->field or item->field->table is null, and suggests adding defensive checks to prevent crashes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

In visit_field, item->field can be NULL if the field is not yet resolved (e.g., during certain error paths or early optimization phases). Additionally, item->field->table could potentially be NULL in some contexts. Dereferencing these pointers without a prior check can lead to a server crash.

To prevent potential null pointer dereferences, please add defensive checks for item->field and item->field->table before accessing their members.

  {
    if (item->field && item->field->table)
    {
      st_select_lex *cmp= new_parent;
      while (cmp && cmp->merged_into)
        cmp= cmp->merged_into;
      if (cmp && item->field->table->pos_in_table_list &&
          (item->field->table->pos_in_table_list->select_lex == cmp))
        used_tables|= item->field->table->map;
      else
        used_tables|= OUTER_REF_TABLE_BIT;
    }
  }