Question: namespace disambiguation rules

[dcastro]

I'm trying to understand how namespace resolution works.
From what I've tested so far:

namespace Foo;
table X {
 field: Bar.Y;
}

In this case, field: Bar.Y could mean one of two things. It can be a reference to a type Y in the nested namespace Foo.Bar.Y if it exists, otherwise it'll be a reference to a type Y in the namespace Bar.

This behaviour seems normal to me. What surprised me was this:

namespace Bar;
table Person {}

namespace Foo.Bar;
struct Person { x: int; }

namespace Foo.Bar;
table Table {
  x: Bar.Person;
}

Here, I expected x: Bar.Person to be a reference to the table Person in the namespace Bar.
But it turns out, it's a reference to the struct in the namespace Foo.Bar.

In fact, there seems to be no way to refer to the table in the namespace Bar.
All of these things end up pointing to the struct: Person, Bar.Person, Foo.Bar.Person.
The table seems to be completely shadowed by the struct, even though they're in different namespaces.

Is this the intended behaviour?
Where can I read more about these rules?

Flatc version: I'm using commit #8f8fb2b3677f55529f126f1adcc70b5dcdcd3290

[aardappel]

Well, there are 2 Bar namespaces here, and it always prefers the one closest to the current namespace (it goes inside-out).

The full details of how name resolution works is,.. well.. this monstrosity:

flatbuffers/src/idl_parser.cpp

Lines 1566 to 1617 in 407fb5d

	StructDef *Parser::LookupCreateStruct(const std::string &name,
	bool create_if_new, bool definition) {
	std::string qualified_name = current_namespace_->GetFullyQualifiedName(name);
	// See if it exists pre-declared by an unqualified use.
	auto struct_def = LookupStruct(name);
	if (struct_def && struct_def->predecl) {
	if (definition) {
	// Make sure it has the current namespace, and is registered under its
	// qualified name.
	struct_def->defined_namespace = current_namespace_;
	structs_.Move(name, qualified_name);
	}
	return struct_def;
	}
	// See if it exists pre-declared by an qualified use.
	struct_def = LookupStruct(qualified_name);
	if (struct_def && struct_def->predecl) {
	if (definition) {
	// Make sure it has the current namespace.
	struct_def->defined_namespace = current_namespace_;
	}
	return struct_def;
	}
	if (!definition) {
	// Search thru parent namespaces.
	for (size_t components = current_namespace_->components.size();
	components && !struct_def; components--) {
	struct_def = LookupStruct(
	current_namespace_->GetFullyQualifiedName(name, components - 1));
	}
	}
	if (!struct_def && create_if_new) {
	struct_def = new StructDef();
	if (definition) {
	structs_.Add(qualified_name, struct_def);
	struct_def->name = name;
	struct_def->defined_namespace = current_namespace_;
	} else {
	// Not a definition.
	// Rather than failing, we create a "pre declared" StructDef, due to
	// circular references, and check for errors at the end of parsing.
	// It is defined in the current namespace, as the best guess what the
	// final namespace will be.
	structs_.Add(name, struct_def);
	struct_def->name = name;
	struct_def->defined_namespace = current_namespace_;
	struct_def->original_location.reset(
	new std::string(file_being_parsed_ + ":" + NumToString(line_)));
	}
	}
	return struct_def;
	}

In particular, here it looks thru all parents, based on the current name:

flatbuffers/src/idl_parser.cpp

Lines 1590 to 1595 in 407fb5d

	// Search thru parent namespaces.
	for (size_t components = current_namespace_->components.size();
	components && !struct_def; components--) {
	struct_def = LookupStruct(
	current_namespace_->GetFullyQualifiedName(name, components - 1));
	}

This will cause it to first look for Foo.Bar.Bar.Person, then Foo.Bar.Person (which it finds), and then Bar.Person would have been next.

I believe that is similar to how C++ works?

[dcastro]

@aardappel Ah that makes sense, thanks for clarifying so promptly! :)

[dcastro]

@aardappel, Would you say that these two specs look correct? Just to make sure I'm understanding things correctly.

namespace ;       enum E1 : short{x}   enum E2 : short{x}   enum E3 : short{x}
namespace A;      enum E1 : short{x}   enum E2 : short{x}
namespace A.B;    enum E1 : short{x}
namespace A.B.C;  enum E1 : short{x}   enum E2 : short{x}   enum E3 : short{x}

namespace A.B;
struct S {
  x: E1; // should be A.B.E1
  y: E2; // should be A.E2
  z: E3; // should be E3

namespace ;         enum E1 : short{x}   enum E2 : short{x}   enum E3 : short{x}
namespace A;        enum E1 : short{x}   enum E2 : short{x}   enum E3 : short{x}
namespace A.B;      enum E1 : short{x}   enum E2 : short{x}   enum E3 : short{x}
namespace A.A;      enum E1 : short{x}   enum E2 : short{x}
namespace A.B.A;    enum E1 : short{x}
namespace A.B.C.A;  enum E1 : short{x}   enum E2 : short{x}   enum E3 : short{x}

namespace A.B;
struct S {
  x: A.E1; // should be A.B.A.E1
  y: A.E2; // should be A.A.E2
  z: A.E3; // should be A.E3
}

[aardappel]

That looks correct to me :)

You're making a Haskell implementation? How cool! Consider contributing it to the main repo when done.

Why are you needing to figure out these namespaces though? You should let flatc handle that, like all the other languages do. Haven't looked at your code yet, but I hope you're not implementing your own schema parser.. making sure you stay in sync with the main one will not be fun.

[dcastro]

@aardappel Apologies for the late reply, I must have missed the notification from github 😮

Yes I am 😄 thank you!

Yeah, I had to implement everything from scratch, as I wanted to use TemplateHaskell (TH) for codegen. TH has two big advantages:

• it's automatically hooked into the compiling process (sorta like java @annotations), so there's no need for additional build steps to run flatc like you would in other languages.
• it's guaranteed to generate syntactically valid code (it doesn't use string concatenation, it has dedicated functions to generate a new data type, a new variable, a new function, etc)

The "downside" is that TemplateHaskell has to be written in, well, Haskell.

I could theoritically call flatc to parse the schema file, do all the semantic validation, and then give me an AST, and then I'd use TemplateHaskell to turn the AST into Haskell code.
But from what I can tell, flatc doesn't expose the AST... so I decided to do all three steps (parsing + semantic validation + codegen with TH) in Haskell

The parsing wasn't that much trouble (~250 lines of mostly one-liners), the semantic validation step was the hard part (~800 lines)...

I was under the impression that flatbuffers was more or less a stable project, so keeping these in sync would be ok-ish, but I'm starting to feel like that's not the case 😮

Is there a better channel to continue this conversation? gitter, maybe?

[aardappel]

That is cool.. in that way it makes sense for Haskell to be a standalone implementation, we can link to it from the FlatBuffers home page.

I guess it is mostly stable at the moment, but the past has seen plenty of changes.

I feel this is a better place for discussion than gitter, since this will be easier to find for people in the future :)

[dcastro]

@aardappel That sounds great, thank you!
I'm also ok with moving the code to the main repo, or keep it separate and link to it, whichever you think is best :)

[aardappel]

The main repo makes most sense if it is integrated with the main codegen and testing, like all the other languages there. Now haskell will have its own codegen, but still could share testing files etc. I'd be ok either way I think. Adding it to the main repo does have the expectation you'll stay involved in maintaining it, answering issues that relate to Haskell etc.

[dcastro]

Adding it to the main repo does have the expectation you'll stay involved in maintaining it, answering issues that relate to Haskell etc.

Of course :)

For now I think I'll focus on getting it to work, hopefully achieve feature parity with the official flatbuffers implementation (might take a while), and then we'll come back to this and decide where to host it :)