Skip to content

Instantly share code, notes, and snippets.

@mrmaheshrajput

mrmaheshrajput/SageMaker jumpstart model deployment.py

Created March 27, 2024 04:22
Show Gist options
  • Save mrmaheshrajput/342ea15db1ab5cf5cfb59d32715b1995 to your computer and use it in GitHub Desktop.
Save mrmaheshrajput/342ea15db1ab5cf5cfb59d32715b1995 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
SageMaker jumpstart model deployment.py
!pip install sagemaker -q
from sagemaker.jumpstart.model import JumpStartModel
model_id, model_version, = (
"huggingface-llm-falcon-7b-instruct-bf16",
"*",
)
my_model = JumpStartModel(model_id=model_id)
example_payloads = my_model.retrieve_all_examples()
predictor = my_model.deploy()
# This may not work
for payload in example_payloads:
response = predictor.predict(payload.body)
print("Input:\n", payload.body[payload.prompt_key])
print("Output:\n", response[0]["generated_text"], "\n\n===============\n")
# Sample
prompt = "Tell me about Amazon SageMaker."
payload = {
"inputs": prompt,
"parameters": {
"do_sample": True,
"top_p": 0.9,
"temperature": 0.8,
"max_new_tokens": 1024,
"stop": ["<|endoftext|>", "</s>"],
},
}
response = predictor.predict(payload)
print(response[0]["generated_text"])
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment