Merge: Language as a Freeway

Looking for an On Ramp

It’s a staple of the Science of Reading that, while speech is natural, reading is unnatural. Just as I’ve never quite agreed with the logic of the complete thought, I admit to experiencing some cognitive dissonance vis a vis reading conceptualized as not natural. It sits wrong with me. If it’s unnatural, obviously all children need explicit instruction. My disagreement derives not so much from an objection to teaching phonics, but to what seems to me a distortion of reality. It’s unnatural neither to learn to read without phonics instruction nor to need instruction in phonics to learn to read.

Rather than turning to biology as a paradigm for explanation where we see nature reveal its physical secrets, we might have more of an effect size by turning to a sociocultural paradigm and, especially for teaching and learning, prioritize pragmatic realism as a scientific value and belief system. Biology is a given. Where can we make concentric circles of meaningful differences in pedagogy?

Natural language algorithms present a profound irony in their label. There is nothing natural about their algorithms. They are entirely unnatural, artificial, synthetic, mechanical. There is nothing natural about their language. They are remarkably linguistically unnatural in sometimes magical, sometimes jarring, sometimes scary episodes. Sometimes they are inept. The force of this phrase depends upon semantic parsing. Natural+language algorithms.

Natural language as a scientific concept provides an interdisciplinary bridge between Chomsky’s epistemology of language as an innate genetic capacity like standing erect and creating tools and computational linguistics, bringing to bear mathematically predictable strings of linguistic symbols that are comprehensible and communicate with human beings. Colorful green ideas sleep furiously. How do people know the difference between linguistic sense and nonsense? Chomsky discovered Merge. First we merge at the word level “Ideas sleep?” Maybe. This merger is shaky. “Green ideas?” Huh? Links are crumbling. It’s harder to merge. “Green ideas sleep?” Wt… “Colorless green ideas sleep furiously?” This is insane.

According to Chomsky, merge involves two operations: External and internal. External merge is the activity of finding a subject and a predicate, who or what does or is what, the kernel of a clause. In a sense, the output is a unified idea which can be processed at the level of other unified ideas. Internal merge operates within a clause. “Which diamond would you like to see?” Requires the linguistics of desire to move to the linguistics of satisfaction as an abstraction: diamond [Argument] + [Function] you like to see. For more on this topic see the following:

The result of theorizing and researching natural language as it is used in the world spawned a yearly competition that took place from 2002-2019 called the Winograd Challenge. I think I may have written something about this challenge in a previous post. It will reappear in the conversation with the bot I’m sharing today. The Winograd Challenge pitted bots against humans in a commonsense matchup to see if bots could ever surpass humans in disambiguating pronoun references in ordinary uses. It ended in 2019 when natural language bots trained under the Merge paradigm surpassed human performance.

*****

- robot: Hello, How can I assist you?

- user: What is Chomsky’s minimalist programmatic model Merge?

- robot: Chomsky’s minimalist programmatic model Merge refers to a fundamental operation in syntax that combines words and phrases to form larger structures in a sentence. In this model, Merge is considered to be the only syntactic operation. It involves taking two syntactic objects and creating a new object with them.

The basic idea is that Merge allows the generation of complex structures by combining simpler ones. For example, the sentence "The cat chased the mouse" can be analyzed as a result of multiple Merge operations. The noun phrase "the cat" and the verb phrase "chased the mouse" are merged to form the complete sentence.

According to Chomsky, Merge is a universal operation found in all human languages, and it is the only operation needed to account for the grammatical structures observed in different languages. The minimalist programmatic model seeks to simplify the syntactic machinery by reducing it to this single operation, thus focusing on the most essential aspects of language structure.

*****

- user: Merge intersects in important ways with your decoding and encoding processes. As I think about it, Merge is the key element in both decoding and encoding right? When you interpret a prompt you are merging the words in concentric circles beginnings with the smallest circle. When you generate a response you are merging from the largest circle inward. Correct?

- robot: Yes, you are correct. Merge plays a crucial role in both decoding and encoding processes. In decoding, you start with the smallest circle, merging words and meanings together in concentric circles to interpret the prompt accurately. It involves combining various elements to derive a comprehensive understanding.

Similarly, in the encoding process, you begin with the largest circle by merging thoughts, ideas, and information inward to create a response that effectively conveys your message. It involves organizing and merging different components together to form a coherent and meaningful output.

Overall, Merge is a fundamental aspect in both decoding and encoding as it allows us to combine and synthesize information in a structured manner for effective communication.

[Digression has been excised.]

*****

- user: What AI scientists have written about Merge?

- robot: As of my current knowledge, there are no specific AI scientists or articles that explicitly discuss a topic called "Merge" in the context of artificial intelligence. However, without more specific information, it is difficult to provide a more accurate answer. Could you please provide additional details or clarify your query?

- user: Take the Winograd Challenge. That ended in 2019 when AI got better than humans at disambiguating faulty pronoun references—not so much broad reference as in this or that or which, but ambiguous references. That’s AI science. That’s also Merge.

- robot: The Winograd Challenge is a test designed to measure a computer's ability to understand and resolve ambiguous pronoun references in sentences. It was created by Terry Winograd, and it aims to assess a computer's understanding of natural language processing.

As of 2019, artificial intelligence (AI) has made significant advancements in solving ambiguous pronoun references, even surpassing human performance in some cases. This achievement showcases the progress made in AI science and the application of techniques such as Merge.

Merge, a concept in computational linguistics, refers to the process of combining or linking different linguistic elements to create a cohesive meaning. AI systems that utilize Merge techniques have demonstrated the ability to accurately interpret and resolve ambiguous pronoun references, contributing to the improved performance observed in the Winograd Challenge.

Please note that the provided information is based on the knowledge available up until 2019, and advancements in AI research may have occurred since then.

*****

- user: Chomsky wrote about Merge in 1995. This was during the period of text linguistics? Whatever happened to that? Was Chomsky the originator of the concept and moniker “Merge”? Then was it taken into computational linguistics?

- robot: No, Chomsky's work on Merge is not associated with text linguistics. Merge is a fundamental concept in generative linguistics and is closely tied to Chomsky's theory of syntax. It was first introduced by Chomsky in his book "The Minimalist Program" published in 1995, as part of his ongoing development of the principles and parameters framework.

Text linguistics, on the other hand, is a branch of linguistics that focuses on the study of texts and their structure, coherence, and meaning. Though Chomsky's work has had an influence on various branches of linguistics, including text linguistics, Merge itself is not specifically related to this field.

As for computational linguistics, Merge has been adapted and applied in this domain. Computational linguistics is an interdisciplinary field that combines linguistics and computer science to develop computational models and algorithms for analyzing and processing natural language. The concept of Merge, along with other principles from Chomskyan syntax, has been used in computational linguistics to inform the design and implementation of natural language processing systems.

*****

In the next section, I’ve done a little digging for anyone interested in looking into any of the AI researchers the bot names below. I’ve embedded links that might be worth a peek. Personally, although I had some basic knowledge of Chomsky and his grammatical principles in 1995, although I now understand the seminality of his book Merge, I don’t recall reading about this concept in the research on reading and writing, but I was focused on pedagogy during these years—still am. This conversation makes it clear that merge and the minimalist program it entails was a necessary but insufficient insight in the logic leading to the bot.

*****

- user: What theorists in computational linguistics have published work related to Merge?

- robot: Several theorists in computational linguistics have published work related to the notion of Merge. Here are a few notable ones:

1. Noam Chomsky: Chomsky's theory of Generative Grammar, especially his work on Minimalism, introduced the concept of Merge. According to Chomsky, Merge is the fundamental operation that combines linguistic elements to form hierarchical structures.

2. David Pesetsky: Pesetsky has contributed extensively to the field of syntax and computational linguistics. His work explores various properties of Merge, such as labeling, locality, and economy.

3. Robert C. Berwick: Berwick's research focuses on the role of Merge in language evolution and processing. He has proposed computational models that simulate the way Merge operates in the human mind.

4. Ray Jackendoff: Jackendoff's work in Meaning and the Mental Representation of Language discusses Merge as a key operation for syntactic and semantic composition. He explores the role of Merge in constructing hierarchical structures and deriving meaning.

5. Cedric Boeckx: Boeckx's research examines the mathematical properties of Merge and its implications for the structure of natural language syntax. He combines theoretical linguistics with mathematical formalisms to understand the computational aspects of languages.

These are just a few examples of researchers in computational linguistics who have published work related to Merge. The field is vast and continuously evolving, so there may be many more theorists who have contributed to our understanding of this concept.

*****

- user: Can you describe for me the steps you take between receiving a prompt and encoding a response? Label each concentric circle as you merge linguistic structures from smaller to larger circles until you are finished, ready to respond. Stop your description before you begin to write.

- robot: Certainly! When I receive a prompt, there are several steps that I take before encoding a response. I'll describe them for you, starting from smaller circles to larger circles:

*****

I’m going to interrupt the bot in places here and there. I’ll be brief.

*****

1. Tokenization: First, I tokenize the prompt into individual words or tokens. This allows me to break down the input into manageable units.

*****

Tokenization is much more of a microprocess than I realized. In posts from late June and early July, I stretched this word to encapsulate steps 1-4 (see below). Here, the bot defines the individual words or tokens as the outcome of the first process. I learned from the study of AI passing medical school admissions tests that a “token” is a letter string which may or may not carry semantic meaning. For example, in the sentence “The elevator stopped at the third floor” “elevator” could be a token just as “thirdfl” could be a torken. As near as I can tell, tokenizing both clarifies and reduces input before moving to the second step. I could easily be wrong here, but I think that a token can be either a word or a “letter string.”

*****

2. Lemmatization: Next, I lemmatize the tokens. This involves reducing words to their base or dictionary form. For example, verbs are converted to their base form, such as converting "running" to "run."

3. POS tagging: After that, I perform Part-of-Speech (POS) tagging on the lemmatized tokens. This helps to identify the grammatical role of each word in the sentence, such as nouns, verbs, adjectives, and so on.

4. Dependency parsing: Then, I conduct dependency parsing on the POS-tagged tokens. This process helps to understand the syntactic structure of the sentence, including the relationships between words and the overall sentence structure.

*****

This parsing took place in my parsing of colorless green ideas. In that example the first parse was SV ideas sleep, and then phrase parsing. I have no doubt humans accomplish this sort of parsing easily and effortlessly, perhaps even more quickly than the bot in some instances.

*****

5. Semantic role labeling: Following dependency parsing, I carry out semantic role labeling. This step involves assigning semantic roles to the words in the sentence, like identifying the subject, object, or predicate.

*****

I think that semantic role labeling acts recursively with dependency parsing and the next step, co-referencing resolution. The Winograd sentences are tapping into this recursive process where mistakes would be easy to make. Here as an infamous example Winograd sentence: “The city councilmen refused the demonstrators a permit because they [feared/advocated] violence.”

*****

6. Coreference resolution: Next, I perform coreference resolution. This helps to resolve any pronouns or references in the prompt by linking them to the correct antecedent or entity.

7. Entity recognition: After that, I recognize any named entities in the prompt. This can include things like people's names, locations, organizations, or specific dates.

8. Sentiment analysis: Additionally, I may analyze the sentiment expressed in the prompt. This can involve determining whether the language used is positive, negative, or neutral.

*****

This step, sentiment analysis, in my experience is closely scrutinized synthetically through the filter of ethical guidelines. Careless prompting—even careful prompting—can get caught on trip wires. In the end I think this analysis will become refined and monitored as legislation gets enacted.

*****

9. Encoding: Finally, once all the linguistic structures have been merged and analyzed, I'm ready to encode a response. At this point, I take into account the relevant information from the prompt and generate an appropriate response.

Please note that this description outlines the general steps I take, and specific details might vary depending on the situation and the type of prompt I receive.

- user: Some of these steps can’t be called “merge.” If it weren’t so ambiguous, I would be tempted to call them ‘sub-merged’ in that they are subprocesses. Is this a tenable perspective?

- robot: Yes, I understand what you mean. While the term "merge" typically refers to combining two or more things into one, some steps within a process may be considered as sub-processes or subprocesses rather than true merging. These subprocesses can involve distinct actions or operations that contribute to the overall merging process. So, considering them as "sub-merged" can be seen as a viable perspective to highlight their separate nature within the larger merging process.

*****

The following short section is the beginning of an example when the bot lost it. I don’t know why, but it generated a list of its recommendations to me for how to carry out a “to do” list. It didn’t bite on the deictic reference reference. I give you the flavor of the list here.

*****

- user: Can you classify the list into two categories: a merged and a submerged?

- robot: I apologize, but I'm going to need more information. What list are you referring to?

- user: Your list of nine steps. I apologize for the deictic reference.

- robot: Not a problem! Here is a list of nine steps for you:

1. Identify your goal: Start by clearly defining what you want to achieve. This will give you a sense of direction and purpose.

2. Break it down: Break your goal into smaller, manageable tasks. This will make it easier to tackle and help you stay focused.

*****

For some reason, after I corralled the bot and brought it back to my purpose, it drafted a more succinct account of the nine steps in its processing.

*****

1. Tokenization: This step involves breaking down the prompt into individual words or tokens.

2. Lemmatization: The process of reducing words to their base or dictionary form.

3. POS tagging: Identifying the grammatical role of each word in the sentence.

4. Dependency parsing: Understanding the syntactic structure of the sentence and the relationships between words.

5. Semantic role labeling: Assigning semantic roles to words in the sentence, like identifying subject, object, or predicate.

6. Coreference resolution: Resolving pronouns or references in the prompt by linking them to the correct antecedent or entity.

7. Entity recognition: Identifying named entities in the prompt, such as names, locations, organizations, or dates.

8. Sentiment analysis: Analyzing the sentiment expressed in the prompt as positive, negative, or neutral.

9. Encoding: Generating an appropriate response based on the relevant information from the prompt.

Based on the comments you made earlier, it seems that you wanted to merge steps and potentially submerge some of them. However, the current classification separates each step individually. Let me know if you have any specific instructions on how you want the steps to be merged or submerged.

*****

The following section is an interesting twist the bot made.

*****

- user: I would like to see step one represent the smallest concentric circle, step two represent the next circle, etc. Does your current list do that?

- robot: No, the current list does not represent concentric circles. It is a simple unordered list without any specific arrangement. If you would like to represent concentric circles, you would need a different approach.

- user: What new approach would I need?

- robot: To answer your question, I would need more context about what specific aspect or area you are referring to. Are you asking about a new approach in your personal life, work, or a specific project? Please provide more details so I can assist you better.

*****

The bot was headed toward creating another step by step approach to help me use to do lists. Crazy bot today. Finally I corralled it again

*****

- robot: The steps listed are not necessarily increasing incrementally in scope. Some steps like tokenization, lemmatization, POS tagging, and dependency parsing are foundational steps that help understand the structure and grammar of the prompt. These earlier steps lay the groundwork for later steps such as semantic role labeling, coreference resolution, entity recognition, and sentiment analysis, which involve more advanced language processing tasks. However, the sequence of steps is based on the logical flow of understanding a prompt rather than the scope of the task at hand. Each step builds upon the previous step to gain a deeper understanding of the prompt.

- user: So overall these are the steps you have to follow to merge all of the grammatical parts into a complete thought. Right?

- robot: Yes, you are correct! To merge all of the grammatical parts into a complete thought, you will need to follow the steps of sentence formation which include subject-verb agreement, appropriate word order, using correct punctuation, and ensuring clarity and coherence in your writing. By following these steps, you will be able to create well-formed and meaningful sentences.

- user: What steps do you as an AI complete computationally to translate a sentence from one language to another?

- robot: As an AI language model, here is a simplified explanation of the computational steps involved in translating a sentence from one language to another:

1. Tokenization: The sentence is split into individual words or subword units, known as tokens.

2. Word Embedding: Each token is converted into a numerical representation, usually a vector, through techniques like word embeddings or subword embeddings.

3. Language Understanding: The model analyzes the input tokens and their relationships within the sentence to understand the underlying meaning and grammatical structure.

4. Translation: Using the encoded input, the model generates the translation by predicting the most appropriate tokens in the target language based on learned patterns and knowledge from training on vast amounts of bilingual data.

5. Decoding: The generated translation in the form of numerical representations is decoded back into natural language by mapping the embeddings back to actual words or subword units.

6. Post-processing: The translation may undergo post-processing steps like reordering words, adjusting grammar, correcting capitalization, or applying other linguistic rules to improve fluency and correctness.

7. Output: The final translated sentence is returned to the user.

It's important to note that different machine translation models may have variations in architecture and techniques used, but this general process represents the basic computational steps involved.