Hi there! I hope you are all keeping well and have started the new year with great enthusiasm and vigor. Let this coming year be a successful one for all of us. Happy New Year 2024!
I have not been able to post anything since the last few weeks as I was taking care of a family emergency situation along with working on other aspects. Thankfully, things are okay now. However, there is a long road ahead for me and my family.
Let's start this year off with some quick information on what happened in the last week before we talk about the model, Mixtral, that I used for testing alongside ChatGPT for a simple coding test. Before that, let’s look at some of the highlights from last week in the world of AI:
Some of last week’s highlights:
Google to cut 30,000 jobs amid AI focus: In a bold move to stay ahead of the curve, Google announced that it would be cutting 30,000 jobs to focus on AI development. The tech giant is looking to invest more in its AI capabilities and stay competitive in the rapidly evolving industry.
Galaxy S24's AI features leaked: Samsung's upcoming flagship phone, the Galaxy S24, has had its AI features leaked. The phone is rumored to have advanced AI capabilities that will revolutionize the way we use our smartphones. Stay tuned for more information as it becomes available.
Get Microsoft Copilot on your phone: Microsoft has announced that its AI-powered Copilot will now be available on mobile devices. This move will allow users to access the powerful AI assistant on the go, making it even easier to stay organized and productive.
AI translates thoughts into text from brainwaves: A groundbreaking new AI technology has been developed that can translate thoughts into text using brainwaves. This innovation has the potential to revolutionize the way we communicate and could have a significant impact on the lives of people with disabilities.
Self-driving cars are safer than human drivers. A recent study found that self-driving cars are now safer than human drivers. The research showed that autonomous vehicles are less likely to be involved in accidents and are more capable of avoiding dangerous situations on the road.
Chief Justice Roberts: Use AI in law 'cautiously': Chief Justice John Roberts has warned that AI should be used 'cautiously' in the legal field. He expressed concerns about the potential for AI to make mistakes and the need for human oversight to ensure that justice is served.
Craft smart New Year's resolutions with AI: Looking to make a change in the new year? AI can help! With advanced algorithms and data analysis, AI can help you craft smart and achievable New Year's resolutions that will set you on the path to success.
A study finds gender bias in AI language models: A new study has found that many AI language models exhibit gender bias. The research highlights the need for more diverse and inclusive training data to ensure that AI is free from bias and can accurately represent the diversity of human language.
Now, let’s get back to the Mixtral and ChatGPT conversations. I was able to run Mixtral on my local machine using a new tool called Jan. Jan is an open-sourced model and is aspiring to support the extensions and make the local LLM studio your own. Download it from here.
I was developing some sample apps with React Native. In the process, I introduced the typo “styles” instead of "style,” and the iOS simulator, when launched through Expo, would not render the screen. There were no errors that I could find right away. Naturally (ahem, we are going to be going to ChatGPT ‘naturally’ now and into the future, probably), I copied and pasted the code into ChatGPT, asking what went wrong with the code. Following is the answer from ChatGPT, which is expected.
However, I wanted to test the same issue with the models I can run locally on my machine. I have tried Phi2, Llama 7B, Mistral 7B, and Mixtral 8x7B 4Q (Q stands for quantized), and 8x is the MoE (Mixture of Experts). The MoE refers to the following:
Mixture of Experts (MoE), in the context of Mixtral AI, is an advanced machine learning concept that involves dividing a complex task into simpler subtasks, each handled by a specialized model known as an expert. Mixtral AI seems to refer to an approach or system that utilizes this MoE model. Here's a brief explanation of the MoE concept:
Multiple Experts: In MoE, the system consists of several "expert" models. Each expert is trained to be proficient in a specific aspect of the overall task. For example, in a language processing system, one expert might focus on grammar, another on semantics, and yet another on the context of a conversation.
Gating Network: A gating network is an essential part of the MoE. It decides which expert to use for a given input. The gating network analyzes the input data and determines which expert (or experts) is best suited to handle it. This decision-making process can be based on the strengths and specialties of each expert.
Combining Outputs: After the experts have processed the input, their outputs are combined to produce the final result. This combination can be a simple aggregation, or it can involve more complex weighting mechanisms based on the confidence or relevance of each expert's output.
Dynamic Specialization: Over time, each expert in the MoE system becomes more specialized in its niche. This dynamic specialization allows the system to handle a wide range of inputs more efficiently, as each type of input is directed to the most competent expert.
Scalability and Efficiency: MoE models are particularly useful in large-scale, complex tasks because they allow for parallel processing and can efficiently handle diverse types of data or problems.
Let's examine the outcome of the same query from Mixtral that I posted to ChatGPT, asking for what went wrong with the code, now that we are familiar with what MoE means. Following is the screenshot.
You can clearly see that the result from Mixtral is almost on par with ChatGPT. I will keep testing these two as I come across other scenarios very specific to coding in the coming few days.
With respect to 2024, I am expecting that this year it will be AI running locally on machines with smaller memory footprints. This is evident from Google’s push for Gemini Nano, along with open-sourced models like Mixtral and Apple’s push into AI with libraries like MLX, etc. This is going to be another interesting year ahead of us. Let’s embrace it, learn, and contribute for the betterment of all. Once again, happy new year to all my readers and contributors!