Ghost 7B Alpha

The large generation of language models focuses on optimizing excellent reasoning, multi-task knowledge, and tools support.

Introduction#

Ghost 7B Alpha is a large language model fine-tuned from Mistral 7B, with a size of 7 billion parameters. The model was developed with the goal of optimizing reasoning ability, multi-task knowledge and supporting tool usage. The model works well with the main trained and optimized languages being English and Vietnamese.

Overall, the model is suitable when making a pretrained version so you can continue to develop the desired tasks, develop virtual assistants, perform features on tasks such as coding, translation, answering questions, creating documents, etc. It is truly an efficient, fast and extremely cheap open model.

Specifications#

• Name: Ghost 7B Alpha.
• Model size: 7 billion parameters.
• Context length: 8K, 8192.
• Languages: English and Vietnamese.
• Main tasks: reasoning, multi-tasking knowledge and function tools.
• License: Ghost 7B LICENSE AGREEMENT.
• Based on: Mistral 7B.
• Distributions: Standard (BF16), GGUF, AWQ.
• Developed by: Ghost X, Hieu Lam.

Links#

• Card model: 🤗 HuggingFace.
• Official website: Ghost 7B Alpha.
• Demo: Playground with Ghost 7B Alpha.

Distributions#

We create many distributions to give you the best access options that best suit your needs. Always make sure you know which version you need and what will help you operate better.

Standard (BF16)#

The standard distribution was used to run the assessments and was found to have the best performance in text generation quality.

GGUF#

GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.

AWQ#

The AWQ algorithm for 4-bit quantization with a 2x speedup during inference.

Prompt / Chat template#

Roles#

Overall, the model supports roles including: system, plugins, user, assistant, execute, response.

• system: Similar to OpenAI GPT4, this will be the place for you to declare context or specify characteristics and expertise for the model.
• user: This is user input, which can be a question or an instruction for the model to follow.
• assistant: The model’s response to a question or instruction received from user.
• plugins: Declare information about modeling tools that are allowed to be used according to predetermined standards.
• excute: Mock iterates over the data type returned by the model for execution. Contains information about tools to use and arguments.
• response: The result after executing the tool, so that the model gives the appropriate answer according to the user’s input.

See the actual code here.

Tips#

• Setting the system prompt in the right context and characteristics will help the model improve performance quite well.
• You can set system prompt to prevent unwanted things.
• In most cases, if you have no idea you can leave the system prompt blank. At this point, the performance will be best for general cases.

Chat template#

Default chat#

In the standard case, without using function tools, only use roles such as system, user and assistant. Conventions are in the following order:

• (1) system
• (2) user
• (3) assistant

Repeat (2) and (3) continuously throughout the conversation.

Chat with tools#

In case of using tools, the conventions are in the following order:

• (1) system

• (2) plugins

• (3) user

• A. When the model finds it inappropriate to use the tool:

  • (4) assistant if the model can answer or is not suitable for the tool. The model proceeds to answer the user’s question.

• B. When the model chooses to use the tool:

  • (4) execute returns the arguments needed to execute the specified tool.
  • (5) response returns the results after applying the tool to the model.
  • (6) assistant the model proceeds to answer the user’s question.

Repeat (3) and (A) or (B) continuously throughout the conversation.

Showcases#

English#

Let’s try a question taken from OpenAI GPT-4 to evaluate the superiority of GPT-4 compared to ChatGPT.

A quiz to see if the model can guess what it’s talking about?!

To this question, I really cannot answer. But the model’s evidence is quite strict. Also, what do you think about this answer?

A reasoning question suddenly popped up during the process of writing an article announcing information about Ghost 7B Alpha. The model gave an impressive answer, at least to its creator.

Try coding skills#

How about asking the model for instructions on creating a Node.js application using Docker for deployment?

mkdir my-project
cd my-project
npm init -y

npm install express --save

const express = require("express");
const app = express();

app.get("/", (req, res) => {
  res.send("Hello World!");
});

app.listen(3000, () => {
  console.log("Server listening on port 3000");
});

FROM node:16

WORKDIR /usr/src/app

COPY package.json .

RUN npm install

COPY . .

EXPOSE 3000

CMD ["node", "server.js"]

docker build -t my-project-image .

docker run -p 3000:3000 my-project-image

Now let’s try asking for a coding model to build a smooth neural network with python.

import keras
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense

model = Sequential()
model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(10, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

model.fit(x_train, y_train, epochs=10)

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense

model = Sequential()
model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(10, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

model.train(x_train, y_train, epochs=10)

Let’s try asking the model how to overcome the “hallucination” phenomenon in A.I development and give context for the model to provide a solution.

Vietnamese#

Try asking the model what it thinks about a famous quote to see if the answer feels natural.

Explore the model’s literary understanding by providing meaningful story prompts, asking to identify what the story is.

The ability to reason to the answer based on the characteristics of is provided.

Try coding skills#

Try asking the Typescript coding model to perform finding the second largest number in the array to see if it works?

/**
 * Tìm số lớn thứ hai trong mảng.
 *
 * @param {number[]} arr Mảng các số nguyên.
 * @returns {number} Số lớn thứ hai trong mảng.
 */
function findSecondLargest(arr: number[]): number {
  // Kiểm tra xem mảng có ít nhất 2 phần tử không.
  if (arr.length < 2) {
    throw new Error('Mảng phải có ít nhất 2 phần tử.');
  }

  // Tìm số lớn nhất và số lớn thứ hai trong mảng.
  let largest = arr[0];
  let secondLargest = Number.MIN_SAFE_INTEGER;
  for (let i = 1; i < arr.length; i++) {
    if (arr[i] > largest) {
      secondLargest = largest;
      largest = arr[i];
    } else if (arr[i] > secondLargest && arr[i] != largest) {
      secondLargest = arr[i];
    }
  }

  // Trả về số lớn thứ hai.
  return secondLargest;
}

Now let’s try asking for a coding model to build a smooth neural network with python.

import keras.models

model = keras.models.Sequential()
model.add(keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 3)))
model.add(keras.layers.MaxPooling2D((2, 2)))
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(128, activation='relu'))
model.add(keras.layers.Dense(2, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=10, validation_data=(x_test, y_test))

import tensorflow as tf

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 3)))
model.add(tf.keras.layers.MaxPooling2D((2, 2)))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128, activation='relu'))
model.add(tf.keras.layers.Dense(2, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=10, validation_data=(x_test, y_test))

Try asking the model about common errors when deploying and using K8S to see what results the model gives.

Tools / Function calls#

The model is trained to be able to choose the right tools to use and to provide useful and accurate answers to the questions it receives. In this example, the model is asked for some information about Apple stock.

Explain#

This is the standard for declaring model tools, in json format. Also, See more about the model’s prompt / chat template.

[
  {
    "type": "function",
    "function": {
      "name": "get_stock_info",
      "description": "Get stock information",
      "parameters": {
        "type": "object",
        "properties": {
          "ticker": {
            "type": "string",
            "description": "Stock ticker",
            "required": true
          }
        }
      }
    }
  }
]

Workflow#

Turn 1#

To use the tools, the model’s supporting role is plugin which includes the declared tooling schema. Below is an example of messages with turn 1 to use.

# A question about Apple stocks in English language.
question = "What is Apple's market capitalization today?"
# It's still the same question but in Vietnamese languae.
# question = "Vốn hoá thị trường Apple hôm nay là bao nhiêu ?"

plugins = [
  {
    "type": "function",
    "function": {
      "name": "get_stock_info",
      "description": "Get stock information",
      "parameters": {
        "type": "object",
        "properties": {
          "ticker": {
            "type": "string",
            "description": "Stock ticker",
            "required": True,
          }
        },
      },
    },
  }
]

messages = [
  {"role": "system", "content": ""},
  {"role": "plugins", "content": json.dumps(plugins, ensure_ascii=False)},
  {"role": "user", "content": question},
]

output = generate_sample(
  messages=messages,
  max_new_tokens=256, temperature=0.4, top_k=50, top_p=1,
)

At this point, the model will return output as the primitive value as follows:

[execute]({"type": "function", "function": "get_stock_info", "arguments": {"ticker": "AAPL"}})

Use the get_tools function to parse the readable and usable results.

{
  "type": "function",
  "function": "get_stock_info",
  "arguments": {
    "ticker": "AAPL"
  }
}

Turn 2#

Get information about tools to use with arguments. Use it to execute and return data, here we will assume the return data is as follows, sampled from polygon.io (cleaned up):

{
  "request_id": "1ffbab9cc78c7addbd8deaab05bdaf6d",
  "results": {
    "ticker": "AAPL",
    "name": "Apple Inc.",
    "market_cap": 2620023949270,
    "description": "Apple is among the largest companies in the world, with a broad portfolio of hardware and software products targeted at consumers and businesses. Apple's iPhone makes up a majority of the firm sales, and Apple's other products like Mac, iPad, and Watch are designed around the iPhone as the focal point of an expansive software ecosystem. Apple has progressively worked to add new applications, like streaming video, subscription bundles, and augmented reality. The firm designs its own software and semiconductors while working with subcontractors like Foxconn and TSMC to build its products and chips. Slightly less than half of Apple's sales come directly through its flagship stores, with a majority of sales coming indirectly through partnerships and distribution.",
    "ticker_root": "AAPL",
    "homepage_url": "https://www.apple.com",
    "total_employees": 161000,
    ...
  },
  "status": "OK"
}

Continue using the model to answer user questions based on the data returned from the tool.

executes = {
  "type": "function",
  "function": "get_stock_info",
  "arguments": {
    "ticker": "AAPL"
  }
}

results = {
  "request_id": "1ffbab9cc78c7addbd8deaab05bdaf6d",
  "results": {
    "ticker": "AAPL",
    "name": "Apple Inc.",
    "market_cap": 2620023949270,
    "description": "Apple is among the largest companies in the world, with a broad portfolio of hardware and software products targeted at consumers and businesses. Apple's iPhone makes up a majority of the firm sales, and Apple's other products like Mac, iPad, and Watch are designed around the iPhone as the focal point of an expansive software ecosystem. Apple has progressively worked to add new applications, like streaming video, subscription bundles, and augmented reality. The firm designs its own software and semiconductors while working with subcontractors like Foxconn and TSMC to build its products and chips. Slightly less than half of Apple's sales come directly through its flagship stores, with a majority of sales coming indirectly through partnerships and distribution.",
    "ticker_root": "AAPL",
    "homepage_url": "https://www.apple.com",
    "total_employees": 161000,
  },
  "status": "OK"
}

messages = [
  {"role": "system", "content": ""},
  {"role": "plugins","content": json.dumps(plugins, ensure_ascii=False)},
  {"role": "user","content": question},
  # Keep the value of `messages` from before, just add new items below.
  {"role": "execute","content": json.dumps(executes, ensure_ascii=False)},
  {"role": "response","content": json.dumps(results, ensure_ascii=False)},
]

output = generate_sample(
  messages=messages,
  max_new_tokens=256, temperature=0.4, top_k=50, top_p=1,
)

The value of output returned by the model will now be similar to the following.

For the question is English.

Apple's market capitalization today is $2.62 trillion.

And the question is Vietnamese.

Vốn hoá thị trường của Apple hôm nay là **2.620 tỷ đô la**. 💰

Usage#

The model is trained to serve well and balance general tasks, so there will be many approaches when used for production.

Things to know#

Recommendations#

General tasks#

Experience it to find the right fit for your own desires. Below are some suggestions that you might consider.

Answer the questions#

In this task, the model can be integrated to develop RAG products to help answer questions for customer care departments, questions about internal documents, policies, product information, order details, etc.

Coding support#

In this task, the model can help find errors or write algorithms and functions that help speed up the domain development process and improve product quality.

Content creation#

In this task, the model can give suggestions for building layouts and content samples that need high creativity and personality in the content.

Finance support#

In this task, the model can assist users in providing suggestions for consolidation or reference on categories such as stocks, derivatives or commodities based on technical indicators and other relevant information. relate to. Of course, the model has been trained on a lot of financial data, but in the end we should remember, use it as a reference and be responsible for your own decisions.

Other#

In addition, the model can also be expanded in many fields such as medicine and health, education and learning,… Take advantage of the model for good and positive things.

Don’t limit your creativity to the model. Feel free to experience it and use it to build great things.

Virtual assistant#

An understanding of the language along with the overall capabilities of the model, may be suitable for integrating its development as a set of one or more tasks in the development of an intelligent virtual assistant.

The model is trained to understand and empathize emotionally, making the response more natural and understanding. This also helps a lot with this request.

As a pre-training#

The model can be further trained to serve the desired tasks better and more effectively if it cannot satisfy the given requirements. Taking advantage of the model’s current superior inference and multi-tasking capabilities, training it for a few separate tasks will help save a lot of effort, time and cost for the project.

Revealed, the model has been integrated to replace some tasks that were previously used by OpenAI GPT4 in the business/product that the founder has been working on. For those tasks, the model is considered effective compared to the cost and is very worthy.

Bias and limitations#

In addition, there are some things you need to know before using as follows:

Generation configuration#

The temperature affects the truth of the answer. Setting a temperature value greater than 0.2 - 0.4 will result in a more creative answer but may affect the accuracy of the answer, please consider this based on your task.

Hint: you can write a prompt to receive input and ask the model to choose the appropriate temperature based on the question, useful in the case of virtual assistant development.

Emotion bias and empathy#

The model is built with an emotional bias and takes positions when faced with problems or reasoning questions. This can be good or bad depending on the context. If you don’t want the model to be rigid in asking certain questions or problems, set up rules in the system prompt for it.

This bias often appears when the model’s system prompt sets it up as an assistant and mentions that it needs to have empathy for/with a certain user/experience. This is intentional during the model training process, it does not come from something unforeseen.

Directly#

To use the model directly, there are many ways to get started, choose one of the following ways to experience it.

Transformers#

For direct use with transformers, you can easily get started with the following steps.

• Firstly, you need to install transformers via the command below with pip.

  pip install -U transformers

• Right now, you can start using the model directly.

  import torch
  from transformers import (
      AutoModelForCausalLM,
      AutoTokenizer,
  )
  
  base_model = "ghost-x/ghost-7b-alpha"
  model = AutoModelForCausalLM.from_pretrained(
      base_model,
      torch_dtype=torch.bfloat16,
      device_map="auto",
  )
  tokenizer = AutoTokenizer.from_pretrained(base_model)
  
  messages = [
      {"role": "system", "content": ""},
      {"role": "user", "content": "Why is the sky blue ?"},
      # {"role": "user", "content": "Tại sao bầu trời lại màu xanh ?"},
  ]
  prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
  inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
  for k,v in inputs.items():
      inputs[k] = v.cuda()
  outputs = model.generate(**inputs, max_new_tokens=512, do_sample=True, top_k=50, top_p=0.95, temperature=0.4)
  results = tokenizer.batch_decode(outputs)[0]
  print(results)

• Additionally, you can also use a model with 4bit quantization to reduce the required resources at least. You can start with the code below.

  import torch
  from transformers import (
      AutoModelForCausalLM,
      AutoTokenizer,
      BitsAndBytesConfig,
  )
  
  base_model = "ghost-x/ghost-7b-alpha"
  bnb_config = BitsAndBytesConfig(
      load_in_4bit=True,
      bnb_4bit_quant_type="nf4",
      bnb_4bit_compute_dtype=torch.bfloat16,
      bnb_4bit_use_double_quant=False,
  )
  model = AutoModelForCausalLM.from_pretrained(
      base_model,
      quantization_config=bnb_config,
      device_map="auto",
  )
  tokenizer = AutoTokenizer.from_pretrained(base_model)
  
  messages = [
      {"role": "system", "content": ""},
      {"role": "user", "content": "Why is the sky blue ?"},
      # {"role": "user", "content": "Tại sao bầu trời lại màu xanh ?"},
  ]
  prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
  inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
  for k,v in inputs.items():
      inputs[k] = v.cuda()
  outputs = model.generate(**inputs, max_new_tokens=512, do_sample=True, top_k=50, top_p=0.95, temperature=0.4)
  results = tokenizer.batch_decode(outputs)[0]
  print(results)

Unsloth#

For direct use with unsloth, you can easily get started with the following steps.

• Firstly, you need to install unsloth via the command below with pip.

  import torch
  major_version, minor_version = torch.cuda.get_device_capability()
  # Must install separately since Colab has torch 2.2.1, which breaks packages
  !pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
  if major_version >= 8:
      # Use this for new GPUs like Ampere, Hopper GPUs (RTX 30xx, RTX 40xx, A100, H100, L40)
      !pip install --no-deps packaging ninja einops flash-attn xformers trl peft accelerate bitsandbytes
  else:
      # Use this for older GPUs (V100, Tesla T4, RTX 20xx)
      !pip install --no-deps xformers trl peft accelerate bitsandbytes
  pass

• Initialize and optimize the model before use.

  from unsloth import FastLanguageModel
  import os, torch, pprint
  
  base_model = "ghost/ghost-7b-alpha"
  model, tokenizer = FastLanguageModel.from_pretrained(
      model_name = base_model,
      max_seq_length = 8192,
      dtype = None,
      load_in_4bit = False, # Change to `True` if you want to use 4bit quantization.
  )
  FastLanguageModel.for_inference(model)

• Right now, you can start using the model directly.

  messages = [
      {"role": "system", "content": ""},
      {"role": "user", "content": "Why is the sky blue ?"},
      # {"role": "user", "content": "Tại sao bầu trời lại màu xanh ?"},
  ]
  prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
  inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
  for k,v in inputs.items():
      inputs[k] = v.cuda()
  outputs = model.generate(**inputs, max_new_tokens=512, do_sample=True, top_k=50, top_p=0.95, temperature=0.4)
  results = tokenizer.batch_decode(outputs)[0]
  print(results)

Online#

The content of this document will be updated soon. The documentation will guide usage through providers such as Huggingface Inference API, Together AI, or Replicate.

Playground#

To make it easy for everyone to quickly experience the model through platforms like Google Colab and Kaggle. We’ve made these notebooks available so you can get started right away. See available notebooks with details here.

We offer notebooks with a wide selection of tools and platforms so you can easily get started. Here are some available options for you to try right now. Let’s pick one to get started.

There are currently two language versions: English and Vietnamese. In each version, it will be almost the same. The content will include installation instructions and functions to help get started quickly with experiencing the model’s text generation capabilities. In addition, it also provides examples with system prompts for each field so you can better understand them.

Playground using Transformers#

• Kaggle @ Ghost 7B Alpha - Playground using Transformers, Vietnamese.

• Colab @ Ghost 7B Alpha - Playground using Transformers, Vietnamese.

• Kaggle @ Ghost 7B Alpha - Playground using Transformers, English.

• Colab @ Ghost 7B Alpha - Playground using Transformers, English.

Playground using Unsloth#

• Kaggle @ Ghost 7B Alpha - Playground using Unsloth, Vietnamese.

• Colab @ Ghost 7B Alpha - Playground using Unsloth, Vietnamese.

• Kaggle @ Ghost 7B Alpha - Playground using Unsloth, English.

• Colab @ Ghost 7B Alpha - Playground using Unsloth, English.

Revealing, experience with Kaggle will give you much better results in creating documents than Colab. We recommend using Kaggle.

Deployments#

The models developed by Ghost X have the same goal of being easy to integrate and use in practice to save costs and facilitate development for the community and startups.

For production deployment with small to large infrastructure, please see more detailed instructions in this article. The article will provide the most common and effective deployment solutions with leading, trusted libraries such as vLLM and more. In addition, it also has information about more optimal solutions and methods depending on each need to be able to choose the appropriate solution.

Evaluation#

To ensure effective model development, we ran popular evaluations to give everyone a broader view of the model’s capabilities. We had to work to balance the quality between the English and Vietnamese languages to ensure the model still performed well in both of these primary languages.

In addition, if your goals are oriented towards a certain main language, you can refer to the advanced user guide to improve model quality for the tasks and languages you desire.

Open LLM Leaderboard#

The results of this evaluation will be updated soon.

MT-Bench#

MT-bench is a challenging multi-turn question set designed to evaluate the conversational and instruction-following ability of models. [source from lmsys.org]

Ghost 7B Alpha achieved a decent score for the MT-Bench review, we worked hard to balance the reasoning ability and linguistic insight of both primary languages, English and Vietnamese. Overall, it was able to outperform some large language models such as tulu-30b, guanaco-65b, and mpt-30b-chat which are many times larger.

AlpacaEval#

AlpacaEval is an automated tool for evaluating instruction-following language models against the AlpacaFarm dataset. It stands out for its human-validated, high-quality assessments that are both cost-effective and rapid. [source from klu.ai]

Similarly, in this evaluation the model achieved good performance as it outperformed larger sized models such as vicuna-13b, guanaco-33b, and nous-hermes-13b.

Notes#

Thanks#

After all, the project also wants to send sincere thanks to the products that have supported the project’s development.

Unsloth#

Unsloth, a great tool that helps us easily develop products, at a lower cost than expected.

Kaggle#

Kaggle, free GPUs are available to the community for research.

And…#

Good friends, who have accompanied the project, Luan Nguyen and Phu Tran.

Confidence#

In addition to the Ghost 7B Alpha project, Ghost X always wants to develop and improve many better models in the future, better supporting the community and businesses with the most openness possible.

Revealing the Ghost 7B Beta project plan. This model is expected to outperform with a deeper focus on multi-tasking, math, and reasoning. Along with that is the ability to expand context length and support other languages (highly requested languages).

The organization is being operated and developed by Hieu Lam’s personal resources, if there is any development support or consulting request. Please feel free to contact the organization, we are very happy about this. Directly via email: .

Ghost X is happy to support providing models for server providers, aiming to help startups develop better.

Contact#

Follow Ghost X to stay updated with the latest information.

• Twitter/X via @ghostx_ai.
• HuggingFace via @ghost-x.
• Official website ghost-x.org.